Re: [PATCH v10 00/41] btrfs: zoned block device support

[Anand Jain <anand.jain@oracle.com>]

On 10/11/20 7:26 pm, Naohiro Aota wrote:
> This series adds zoned block device support to btrfs.
> 
> This series is also available on github.
> Kernel   https://github.com/naota/linux/tree/btrfs-zoned-v10

  This branch is not reachable. Should it be

     https://github.com/naota/linux/tree/btrfs-zoned-for-v10 ?

  But the commits in this branch are at a pre-fixups stage.

Thanks, Anand


> Userland https://github.com/naota/btrfs-progs/tree/btrfs-zoned
> xfstests https://github.com/naota/fstests/tree/btrfs-zoned

> 
> Userland tool depends on patched util-linux (libblkid and wipefs) to handle
> log-structured superblock. To ease the testing, pre-compiled static linked
> userland tools are available here:
> https://wdc.app.box.com/s/fnhqsb3otrvgkstq66o6bvdw6tk525kp
> 
> This v10 still leaves the following issues left for later fix. But, the
> first part of the series should be good shape to be merged.
> - Bio submission path & splitting an ordered extent
> - Redirtying freed tree blocks
>    - Switch to keeping it dirty
>      - Not working correctly for now
> - Dedicated tree-log block group
>    - We need tree-log for zoned device
>      - Dbench (32 clients) is 85% slower with "-o notreelog"
>    - Need to separate tree-log block group from other metadata space_info
> - Relocation
>    - Use normal write command for relocation
>    - Relocated device extents must be reset
>      - It should be discarded on regular btrfs too though
> 
> Changes from v9:
>    - Extract iomap_dio_bio_opflags() to set the proper bi_opf flag
>    - write pointer emulation
>      - Rewrite using btrfs_previous_extent_item()
>      - Convert ASSERT() to runtime check
>    - Exclude regular superblock positions
>    - Fix an error on writing to conventional zones
>    - Take the transaction lock in mark_block_group_to_copy()
>    - Rename 'hmzoned_devices' to 'zoned_devices' in btrfs_check_zoned_mode()
>    - Add do_discard_extent() helper
>    - Move zoned check into fetch_cluster_info()
>    - Drop setting bdev to bio in btrfs_bio_add_page() (will fix later once
>      we support multiple devices)
>    - Subtract bytes_zone_unusable properly when removing a block group
>    - Add "struct block_device *bdev" directly to btrfs_rmap_block()
>    - Rename btrfs_zone_align to btrfs_align_offset_to_zone
>    - Add comment to use pr_info in place of btrfs_info
>    - Add comment for superblock log zones
>    - Fix coding style
>    - Fix typos
> 
> btrfs-progs and xfstests series will follow.
> 
> This version of ZONED btrfs switched from normal write command to zone
> append write command. You do not need to specify LBA (at the write pointer)
> to write for zone append write command. Instead, you only select a zone to
> write with its start LBA. Then the device (NVMe ZNS), or the emulation of
> zone append command in the sd driver in the case of SAS or SATA HDDs,
> automatically writes the data at the write pointer position and return the
> written LBA as a command reply.
> 
> The benefit of using the zone append write command is that write command
> issuing order does not matter. So, we can eliminate block group lock and
> utilize asynchronous checksum, which can reorder the IOs.
> 
> Eliminating the lock improves performance. In particular, on a workload
> with massive competing to the same zone [1], we observed 36% performance
> improvement compared to normal write.
> 
> [1] Fio running 16 jobs with 4KB random writes for 5 minutes
> 
> However, there are some limitations. We cannot use the non-SINGLE profile.
> Supporting non-SINGLE profile with zone append writing is not trivial. For
> example, in the DUP profile, we send a zone append writing IO to two zones
> on a device. The device reply with written LBAs for the IOs. If the offsets
> of the returned addresses from the beginning of the zone are different,
> then it results in different logical addresses.
> 
> For the same reason, we cannot issue multiple IOs for one ordered extent.
> Thus, the size of an ordered extent is limited under max_zone_append_size.
> This limitation will cause fragmentation and increased usage of metadata.
> In the future, we can add optimization to merge ordered extents after
> end_bio.
> 
> * Patch series description
> 
> A zoned block device consists of a number of zones. Zones are either
> conventional and accepting random writes or sequential and requiring
> that writes be issued in LBA order from each zone write pointer
> position. This patch series ensures that the sequential write
> constraint of sequential zones is respected while fundamentally not
> changing BtrFS block and I/O management for block stored in
> conventional zones.
> 
> To achieve this, the default chunk size of btrfs is changed on zoned
> block devices so that chunks are always aligned to a zone. Allocation
> of blocks within a chunk is changed so that the allocation is always
> sequential from the beginning of the chunks. To do so, an allocation
> pointer is added to block groups and used as the allocation hint.  The
> allocation changes also ensure that blocks freed below the allocation
> pointer are ignored, resulting in sequential block allocation
> regardless of the chunk usage.
> 
> The zone of a chunk is reset to allow reuse of the zone only when the
> block group is being freed, that is, when all the chunks of the block
> group are unused.
> 
> For btrfs volumes composed of multiple zoned disks, a restriction is
> added to ensure that all disks have the same zone size. This
> restriction matches the existing constraint that all chunks in a block
> group must have the same size.
> 
> * Enabling tree-log
> 
> The tree-log feature does not work on ZONED mode as is. Blocks for a
> tree-log tree are allocated mixed with other metadata blocks, and btrfs
> writes and syncs the tree-log blocks to devices at the time of fsync(),
> which is different timing than a global transaction commit. As a result,
> both writing tree-log blocks and writing other metadata blocks become
> non-sequential writes which ZONED mode must avoid.
> 
> This series introduces a dedicated block group for tree-log blocks to
> create two metadata writing streams, one for tree-log blocks and the
> other for metadata blocks. As a result, each write stream can now be
> written to devices separately and sequentially.
> 
> * Log-structured superblock
> 
> Superblock (and its copies) is the only data structure in btrfs which
> has a fixed location on a device. Since we cannot overwrite in a
> sequential write required zone, we cannot place superblock in the
> zone.
> 
> This series implements superblock log writing. It uses two zones as a
> circular buffer to write updated superblocks. Once the first zone is filled
> up, start writing into the second zone. The first zone will be reset once
> both zones are filled. We can determine the postion of the latest
> superblock by reading the write pointer information from a device.
> 
> * Patch series organization
> 
> Patches 1 and 2 are preparing patches for block and iomap layer.
> 
> Patch 3 introduces the ZONED incompatible feature flag to indicate that the
> btrfs volume was formatted for use on zoned block devices.
> 
> Patches 4 to 6 implement functions to gather information on the zones of
> the device (zones type, write pointer position, and max_zone_append_size).
> 
> Patches 7 to 10 disable features which are not compatible with the
> sequential write constraints of zoned block devices. These includes
> space_cache, NODATACOW, fallocate, and MIXED_BG.
> 
> Patch 11 implements the log-structured superblock writing.
> 
> Patches 12 and 13 tweak the device extent allocation for ZONED mode and add
> verification to check if a device extent is properly aligned to zones.
> 
> Patches 14 to 17 implements sequential block allocator for ZONED mode.
> 
> Patch 18 implement a zone reset for unused block groups.
> 
> Patches 19 to 30 implement the writing path for several types of IO
> (non-compressed data, direct IO, and metadata). These include re-dirtying
> once-freed metadata blocks to prevent write holes.
> 
> Patches 31 to 40 tweak some btrfs features work with ZONED mode. These
> include device-replace, relocation, repairing IO error, and tree-log.
> 
> Finally, patch 41 adds the ZONED feature to the list of supported features.
> 
> * Patch testing note
> 
> ** Zone-aware util-linux
> 
> Since the log-structured superblock feature changed the location of
> superblock magic, the current util-linux (libblkid) cannot detect ZONED
> btrfs anymore. You need to apply a to-be posted patch to util-linux to make
> it "zone aware".
> 
> ** Testing device
> 
> You need devices with zone append writing command support to run ZONED
> btrfs.
> 
> Other than real devices, null_blk supports zone append write command. You
> can use memory backed null_blk to run the test on it. Following script
> creates 12800 MB /dev/nullb0.
> 
>      sysfs=/sys/kernel/config/nullb/nullb0
>      size=12800 # MB
>      
>      # drop nullb0
>      if [[ -d $sysfs ]]; then
>              echo 0 > "${sysfs}"/power
>              rmdir $sysfs
>      fi
>      lsmod | grep -q null_blk && rmmod null_blk
>      modprobe null_blk nr_devices=0
>      
>      mkdir "${sysfs}"
>      
>      echo "${size}" > "${sysfs}"/size
>      echo 1 > "${sysfs}"/zoned
>      echo 0 > "${sysfs}"/zone_nr_conv
>      echo 1 > "${sysfs}"/memory_backed
>      
>      echo 1 > "${sysfs}"/power
>      udevadm settle
> 
> Zoned SCSI devices such as SMR HDDs or scsi_debug also support the zone
> append command as an emulated command within the SCSI sd driver. This
> emulation is completely transparent to the user and provides the same
> semantic as a NVMe ZNS native drive support.
> 
> Also, there is a qemu patch available to enable NVMe ZNS device.
> 
> ** xfstests
> 
> We ran xfstests on ZONED btrfs, and, if we omit some cases that are known
> to fail currently, all test cases pass.
> 
> Cases that can be ignored:
> 1) failing also with the regular btrfs on regular devices,
> 2) trying to test fallocate feature without testing with
>     "_require_xfs_io_command "falloc"",
> 3) trying to test incompatible features for ZONED btrfs (e.g. RAID5/6)
> 4) trying to use incompatible setup for ZONED btrfs (e.g. dm-linear not
>     aligned to zone boundary, swap)
> 5) trying to create a file system with too small size, (we require at least
>     9 zones to initiate a ZONED btrfs)
> 6) dropping original MKFS_OPTIONS ("-O zoned"), so it cannot create ZONED
>     btrfs (btrfs/003)
> 7) having ENOSPC which incurred by larger metadata block group size
> 
> I will send a patch series for xfstests to handle these cases (2-6)
> properly.
> 
> Patched xfstests is available here:
> 
> https://github.com/naota/fstests/tree/btrfs-zoned
> 
> Also, you need to apply the following patch if you run xfstests with
> tcmu devices. xfstests btrfs/003 failed to "_devmgt_add" after
> "_devmgt_remove" without this patch.
> 
> https://marc.info/?l=linux-scsi&m=156498625421698&w=2
> 
> v9 https://lore.kernel.org/linux-btrfs/cover.1604065156.git.naohiro.aota@wdc.com/
> v8 https://lore.kernel.org/linux-btrfs/cover.1601572459.git.naohiro.aota@wdc.com/
> v7 https://lore.kernel.org/linux-btrfs/20200911123259.3782926-1-naohiro.aota@wdc.com/
> v6 https://lore.kernel.org/linux-btrfs/20191213040915.3502922-1-naohiro.aota@wdc.com/
> v5 https://lore.kernel.org/linux-btrfs/20191204082513.857320-1-naohiro.aota@wdc.com/
> v4 https://lwn.net/Articles/797061/
> v3 https://lore.kernel.org/linux-btrfs/20190808093038.4163421-1-naohiro.aota@wdc.com/
> v2 https://lore.kernel.org/linux-btrfs/20190607131025.31996-1-naohiro.aota@wdc.com/
> v1 https://lore.kernel.org/linux-btrfs/20180809180450.5091-1-naota@elisp.net/
> 
> Changelog
> v9
>   - Direct-IO path now follow several hardware restrictions (other than
>     max_zone_append_size) by using ZONE_APPEND support of iomap
>   - introduces union of fs_info->zone_size and fs_info->zoned [Johannes]
>     - and use btrfs_is_zoned(fs_info) in place of btrfs_fs_incompat(fs_info, ZONED)
>   - print if zoned is enabled or not when printing module info [Johannes]
>   - drop patch of disabling inode_cache on ZONED
>   - moved for_teelog flag to a proper location [Johannes]
>   - Code style fixes [Johannes]
>   - Add comment about adding physical layer things to ordered extent
>     structure
>   - Pass file_offset explicitly to extract_ordered_extent() instead of
>     determining it from bio
>   - Bug fixes
>     - write out fsync region so that the logical address of ordered extents
>       and checksums are properly finalized
>     - free zone_info at umount time
>     - fix superblock log handling when entering zones[1] in the first time
>     - fixes double free of log-tree roots [Johannes]
>     - Drop erroneous ASSERT in do_allocation_zoned()
> v8
>   - Use bio_add_hw_page() to build up bio to honor hardware restrictions
>     - add bio_add_zone_append_page() as a wrapper of the function
>   - Split file extent on submitting bio
>     - If bio_add_zone_append_page() fails, split the file extent and send
>       out bio
>     - so, we can ensure one bio == one file extent
>   - Fix build bot issues
>   - Rebased on misc-next
> v7:
>   - Use zone append write command instead of normal write command
>     - Bio issuing order does not matter
>     - No need to use lock anymore
>     - Can use asynchronous checksum
>   - Removed RAID support for now
>   - Rename HMZONED to ZONED
>   - Split some patches
>   - Rebased on kdave/for-5.9-rc3 + iomap direct IO
> v6:
>   - Use bitmap helpers (Johannes)
>   - Code cleanup (Johannes)
>   - Rebased on kdave/for-5.5
>   - Enable the tree-log feature.
>   - Treat conventional zones as sequential zones, so we can now allow
>     mixed allocation of conventional zone and sequential write required
>     zone to construct a block group.
>   - Implement log-structured superblock
>     - No need for one conventional zone at the beginning of a device.
>   - Fix deadlock of direct IO writing
>   - Fix building with !CONFIG_BLK_DEV_ZONED (Johannes)
>   - Fix leak of zone_info (Johannes)
> v5:
>   - Rebased on kdave/for-5.5
>   - Enable the tree-log feature.
>   - Treat conventional zones as sequential zones, so we can now allow
>     mixed allocation of conventional zone and sequential write required
>     zone to construct a block group.
>   - Implement log-structured superblock
>     - No need for one conventional zone at the beginning of a device.
>   - Fix deadlock of direct IO writing
>   - Fix building with !CONFIG_BLK_DEV_ZONED (Johannes)
>   - Fix leak of zone_info (Johannes)
> v4:
>   - Move memory allcation of zone informattion out of
>     btrfs_get_dev_zones() (Anand)
>   - Add disabled features table in commit log (Anand)
>   - Ensure "max_chunk_size >= devs_min * data_stripes * zone_size"
> v3:
>   - Serialize allocation and submit_bio instead of bio buffering in
>     btrfs_map_bio().
>   -- Disable async checksum/submit in HMZONED mode
>   - Introduce helper functions and hmzoned.c/h (Josef, David)
>   - Add support for repairing IO failure
>   - Add support for NOCOW direct IO write (Josef)
>   - Disable preallocation entirely
>   -- Disable INODE_MAP_CACHE
>   -- relocation is reworked not to rely on preallocation in HMZONED mode
>   - Disable NODATACOW
>   -Disable MIXED_BG
>   - Device extent that cover super block position is banned (David)
> v2:
>   - Add support for dev-replace
>   -- To support dev-replace, moved submit_buffer one layer up. It now
>      handles bio instead of btrfs_bio.
>   -- Mark unmirrored Block Group readonly only when there are writable
>      mirrored BGs. Necessary to handle degraded RAID.
>   - Expire worker use vanilla delayed_work instead of btrfs's async-thread
>   - Device extent allocator now ensure that region is on the same zone type.
>   - Add delayed allocation shrinking.
>   - Rename btrfs_drop_dev_zonetypes() to btrfs_destroy_dev_zonetypes
>   - Fix
>   -- Use SECTOR_SHIFT (Nikolay)
>   -- Use btrfs_err (Nikolay)
> 
> 
> Johannes Thumshirn (1):
>    block: add bio_add_zone_append_page
> 
> Naohiro Aota (40):
>    iomap: support REQ_OP_ZONE_APPEND
>    btrfs: introduce ZONED feature flag
>    btrfs: get zone information of zoned block devices
>    btrfs: check and enable ZONED mode
>    btrfs: introduce max_zone_append_size
>    btrfs: disallow space_cache in ZONED mode
>    btrfs: disallow NODATACOW in ZONED mode
>    btrfs: disable fallocate in ZONED mode
>    btrfs: disallow mixed-bg in ZONED mode
>    btrfs: implement log-structured superblock for ZONED mode
>    btrfs: implement zoned chunk allocator
>    btrfs: verify device extent is aligned to zone
>    btrfs: load zone's alloction offset
>    btrfs: emulate write pointer for conventional zones
>    btrfs: track unusable bytes for zones
>    btrfs: do sequential extent allocation in ZONED mode
>    btrfs: reset zones of unused block groups
>    btrfs: redirty released extent buffers in ZONED mode
>    btrfs: extract page adding function
>    btrfs: use bio_add_zone_append_page for zoned btrfs
>    btrfs: handle REQ_OP_ZONE_APPEND as writing
>    btrfs: split ordered extent when bio is sent
>    btrfs: extend btrfs_rmap_block for specifying a device
>    btrfs: use ZONE_APPEND write for ZONED btrfs
>    btrfs: enable zone append writing for direct IO
>    btrfs: introduce dedicated data write path for ZONED mode
>    btrfs: serialize meta IOs on ZONED mode
>    btrfs: wait existing extents before truncating
>    btrfs: avoid async metadata checksum on ZONED mode
>    btrfs: mark block groups to copy for device-replace
>    btrfs: implement cloning for ZONED device-replace
>    btrfs: implement copying for ZONED device-replace
>    btrfs: support dev-replace in ZONED mode
>    btrfs: enable relocation in ZONED mode
>    btrfs: relocate block group to repair IO failure in ZONED
>    btrfs: split alloc_log_tree()
>    btrfs: extend zoned allocator to use dedicated tree-log block group
>    btrfs: serialize log transaction on ZONED mode
>    btrfs: reorder log node allocation
>    btrfs: enable to mount ZONED incompat flag
> 
>   block/bio.c                       |   38 +
>   fs/btrfs/Makefile                 |    1 +
>   fs/btrfs/block-group.c            |   84 +-
>   fs/btrfs/block-group.h            |   18 +-
>   fs/btrfs/ctree.h                  |   20 +-
>   fs/btrfs/dev-replace.c            |  195 +++++
>   fs/btrfs/dev-replace.h            |    3 +
>   fs/btrfs/disk-io.c                |   93 ++-
>   fs/btrfs/disk-io.h                |    2 +
>   fs/btrfs/extent-tree.c            |  218 ++++-
>   fs/btrfs/extent_io.c              |  130 ++-
>   fs/btrfs/extent_io.h              |    2 +
>   fs/btrfs/file.c                   |    6 +-
>   fs/btrfs/free-space-cache.c       |   58 ++
>   fs/btrfs/free-space-cache.h       |    2 +
>   fs/btrfs/inode.c                  |  164 +++-
>   fs/btrfs/ioctl.c                  |   13 +
>   fs/btrfs/ordered-data.c           |   79 ++
>   fs/btrfs/ordered-data.h           |   10 +
>   fs/btrfs/relocation.c             |   35 +-
>   fs/btrfs/scrub.c                  |  145 ++++
>   fs/btrfs/space-info.c             |   13 +-
>   fs/btrfs/space-info.h             |    4 +-
>   fs/btrfs/super.c                  |   19 +-
>   fs/btrfs/sysfs.c                  |    4 +
>   fs/btrfs/tests/extent-map-tests.c |    2 +-
>   fs/btrfs/transaction.c            |   10 +
>   fs/btrfs/transaction.h            |    3 +
>   fs/btrfs/tree-log.c               |   52 +-
>   fs/btrfs/volumes.c                |  322 +++++++-
>   fs/btrfs/volumes.h                |    7 +
>   fs/btrfs/zoned.c                  | 1272 +++++++++++++++++++++++++++++
>   fs/btrfs/zoned.h                  |  295 +++++++
>   fs/iomap/direct-io.c              |   41 +-
>   include/linux/bio.h               |    2 +
>   include/linux/iomap.h             |    1 +
>   include/uapi/linux/btrfs.h        |    1 +
>   37 files changed, 3246 insertions(+), 118 deletions(-)
>   create mode 100644 fs/btrfs/zoned.c
>   create mode 100644 fs/btrfs/zoned.h
>