[GIT PULL] Btrfs updates for 4.18

[David Sterba <dsterba@suse.com>]

Hi,

there are some new features and a usual load of cleanups, more details below.

Specifically, there's a set of new non-privileged ioctls to allow
subvolume listing.  It works but still needs a security review as it's a
new interface and we might need to do some tweaks to the data
structures. The fixes could be considred regressions but may touch the
interfaces too.

Currently there are no merge conflicts but linux-next has reported a few
in the past, originating from other *FS trees.

Please pull, thanks.

---

User visible features:

- added support for the ioctl FS_IOC_FSGETXATTR, per-inode flags, successor
  of GET/SETFLAGS; now supports only existing flags: append, immutable,
  noatime, nodump, sync

- 3 new unprivileged ioctls to allow users to enumerate subvolumes

- dedupe syscall implementation does not restrict the range to 16MiB, though it
  still splits the whole range to 16MiB chunks

- on user demand, rmdir() is able to delete an empty subvolume, export the
  capability in sysfs

- fix inode number types in tracepoints, other cleanups

- send: improved speed when dealing with a large removed directory,
  measurements show decrease from 2000 minutes to 2 minutes on a directory with
  2 million entries

- pre-commit check of superblock to detect a mysterious in-memory corruption

- log message updates


Other changes:

- orphan inode cleanup improved, does no keep long-standing reservations that
  could lead up to early ENOSPC in some cases

- slight improvement of handling snapshotted NOCOW files by avoiding some
  unnecessary tree searches

- avoid OOM when dealing with many unmergeable small extents at flush time

- speedup conversion of free space tree representations from/to bitmap/tree

- code refactoring, deletion, cleanups
  - delayed refs
  - delayed iput
  - redundant argument removals
  - memory barrier cleanups
  - remove a redundant mutex supposedly excluding several ioctls to run in
    parallel

- new tracepoints for blockgroup manipulation

- more sanity checks of compressed headers

----------------------------------------------------------------
The following changes since commit b04e217704b7f879c6b91222b066983a44a7a09f:

  Linux 4.17-rc7 (2018-05-27 13:01:47 -0700)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-4.18-tag

for you to fetch changes up to 23d0b79dfaed2305b500b0215b0421701ada6b1a:

  btrfs: Add unprivileged version of ino_lookup ioctl (2018-05-31 11:35:24 +0200)

----------------------------------------------------------------
Al Viro (1):
      btrfs: take the last remnants of ->d_fsdata use out

Anand Jain (19):
      btrfs: add comment about BTRFS_FS_EXCL_OP
      btrfs: rename struct btrfs_fs_devices::list
      btrfs: cleanup __btrfs_open_devices() drop head pointer
      btrfs: rename __btrfs_close_devices to close_fs_devices
      btrfs: rename __btrfs_open_devices to open_fs_devices
      btrfs: cleanup find_device() drop list_head pointer
      btrfs: cleanup btrfs_rm_device() promote fs_devices pointer
      btrfs: move btrfs_raid_type_names values to btrfs_raid_attr table
      btrfs: move btrfs_raid_group values to btrfs_raid_attr table
      btrfs: move btrfs_raid_mindev_errorvalues to btrfs_raid_attr table
      btrfs: reduce uuid_mutex critical section while scanning devices
      btrfs: use existing cur_devices, cleanup btrfs_rm_device
      btrfs: document uuid_mutex uasge in read_chunk_tree
      btrfs: replace uuid_mutex by device_list_mutex in btrfs_open_devices
      btrfs: drop uuid_mutex in btrfs_dev_replace_finishing
      btrfs: drop uuid_mutex in btrfs_destroy_dev_replace_tgtdev
      btrfs: use common variable for fs_devices in btrfs_destroy_dev_replace_tgtdev
      btrfs: add prefix "balance:" for log messages
      btrfs: fix describe_relocation when printing unknown flags

Chengguang Xu (1):
      btrfs: return original error code when failing from option parsing

Colin Ian King (1):
      btrfs: send: fix spelling mistake: "send_in_progres" -> "send_in_progress"

David Sterba (38):
      btrfs: tracepoints, use correct type for inode number
      btrfs: tracepoints, use %llu instead of %Lu
      btrfs: tracepoints, drop unnecessary ULL casts
      btrfs: tracepoints, fix whitespace in strings
      btrfs: tracepoints, use extended format with UUID where possible
      btrfs: tests: pass fs_info to extent_map tests
      btrfs: use fs_info for btrfs_handle_em_exist tracepoint
      btrfs: squeeze btrfs_dev_replace_continue_on_mount to its caller
      btrfs: make success path out of btrfs_init_dev_replace_tgtdev more clear
      btrfs: export and rename free_device
      btrfs: move btrfs_init_dev_replace_tgtdev to dev-replace.c and make static
      btrfs: move volume_mutex to callers of btrfs_rm_device
      btrfs: move clearing of EXCL_OP out of __cancel_balance
      btrfs: add proper safety check before resuming dev-replace
      btrfs: add sanity check when resuming balance after mount
      btrfs: cleanup helpers that reset balance state
      btrfs: remove wrong use of volume_mutex from btrfs_dev_replace_start
      btrfs: kill btrfs_fs_info::volume_mutex
      btrfs: track running balance in a simpler way
      btrfs: move and comment read-only check in btrfs_cancel_balance
      btrfs: drop lock parameter from update_ioctl_balance_args and rename
      btrfs: use mutex in btrfs_resume_balance_async
      btrfs: open code set_balance_control
      btrfs: remove redundant btrfs_balance_control::fs_info
      btrfs: introduce conditional wakeup helpers
      btrfs: add barriers to btrfs_sync_log before log_commit_wait wakeups
      btrfs: replace waitqueue_actvie with cond_wake_up
      btrfs: rename btrfs_update_iflags to reflect which flags it touches
      btrfs: rename btrfs_mask_flags to reflect which flags it touches
      btrfs: rename check_flags to reflect which flags it touches
      btrfs: rename btrfs_flags_to_ioctl to reflect which flags it touches
      btrfs: add helpers for FS_XFLAG_* conversion
      btrfs: add FS_IOC_FSGETXATTR ioctl
      btrfs: add FS_IOC_FSSETXATTR ioctl
      btrfs: unify naming of flags variables for SETFLAGS and XFLAGS
      btrfs: use kvzalloc for EXTENT_SAME temporary data
      btrfs: tests: add helper for error messages and update them
      btrfs: tests: drop newline from test_msg strings

Ethan Lien (2):
      btrfs: lift some btrfs_cross_ref_exist checks in nocow path
      btrfs: balance dirty metadata pages in btrfs_finish_ordered_io

Gu JinXiang (2):
      btrfs: drop unused parameter qgroup_reserved
      btrfs: drop useless member qgroup_reserved of btrfs_pending_snapshot

Gu Jinxiang (3):
      btrfs: remove unused fs_info parameter
      btrfs: do reverse path readahead in btrfs_shrink_device
      btrfs: propagate failures of __exclude_logged_extent to upper caller

Howard McLauchlan (3):
      btrfs: clean up le_bitmap_{set, clear}()
      btrfs: optimize free space tree bitmap conversion
      btrfs: remove unused le_test_bit()

Kees Cook (1):
      btrfs: raid56: Remove VLA usage

Liu Bo (7):
      Btrfs: add parent_transid parameter to veirfy_level_key
      Btrfs: remove superfluous free_extent_buffer in read_block_for_search
      Btrfs: use more straightforward extent_buffer_uptodate check
      Btrfs: move get root out of btrfs_search_slot to a helper
      Btrfs: grab write lock directly if write_lock_level is the max level
      Btrfs: remove always true check in unlock_up
      Btrfs: remove unused check of skip_locking

Lu Fengqi (3):
      btrfs: drop unused space_info parameter from create_space_info
      btrfs: Remove fs_info argument from btrfs_uuid_tree_add
      btrfs: Remove fs_info argument from btrfs_uuid_tree_rem

Misono Tomohiro (5):
      btrfs: Move may_destroy_subvol() from ioctl.c to inode.c
      btrfs: Factor out the main deletion process from btrfs_ioctl_snap_destroy()
      btrfs: Allow rmdir(2) to delete an empty subvolume
      btrfs: sysfs: Add entry which shows if rmdir can work on subvolumes
      btrfs: use error code returned by btrfs_read_fs_root_no_name in search ioctl

Nikolay Borisov (54):
      btrfs: Replace owner argument in add_pinned_bytes with a boolean
      btrfs: Drop delayed_refs argument from btrfs_check_delayed_seq
      btrfs: Use while loop instead of labels in __endio_write_update_ordered
      btrfs: Fix lock release order
      btrfs: Consolidate error checking for btrfs_alloc_chunk
      btrfs: Sink extent_tree arguments in try_release_extent_mapping
      btrfs: Remove map argument from try_release_extent_state
      btrfs: Remove redundant tree argument from extent_readpages
      btrfs: Use list_empty instead of list_empty_careful
      btrfs: Remove tree argument from extent_writepages
      btrfs: Remove btrfs_wait_and_free_delalloc_work
      btrfs: Drop add_delayed_ref_head fs_info parameter
      btrfs: Drop fs_info parameter from add_delayed_data_ref
      btrfs: Drop fs_info parameter from btrfs_merge_delayed_refs
      btrfs: Remove delayed_iput parameter of btrfs_start_delalloc_roots
      btrfs: Remove delayed_iput parameter from btrfs_start_delalloc_inodes
      btrfs: Remove delay_iput parameter from __start_delalloc_inodes
      btrfs: Remove delayed_iput member from btrfs_delalloc_work
      btrfs: Unexport btrfs_alloc_delalloc_work
      btrfs: Remove devid parameter from btrfs_rmap_block
      btrfs: Factor out common delayed refs init code
      btrfs: Use init_delayed_ref_common in add_delayed_tree_ref
      btrfs: Use init_delayed_ref_common in add_delayed_data_ref
      btrfs: Open-code add_delayed_tree_ref
      btrfs: Open-code add_delayed_data_ref
      btrfs: Introduce init_delayed_ref_head
      btrfs: Use init_delayed_ref_head in add_delayed_ref_head
      btrfs: split delayed ref head initialization and addition
      btrfs: Add assert in __btrfs_del_delalloc_inode
      btrfs: Make btrfs_init_dummy_trans initialize trans' fs_info field
      btrfs: Remove fs_info argument from add_block_group_free_space
      btrfs: Remove fs_info argument from __add_block_group_free_space
      btrfs: Remove fs_info argument from __add_to_free_space_tree
      btrfs: Remove fs_info parameter from add_new_free_space_info
      btrfs: Remove fs_info argument from add_new_free_space
      btrfs: Remove fs_info parameter from remove_block_group_free_space
      btrfs: Remove fs_info argument from convert_free_space_to_bitmaps
      btrfs: Remove fs_info parameter from convert_free_space_to_extents
      btrfs: Remove fs_info argument from update_free_space_extent_count
      btrfs: Remove fs_info argument from modify_free_space_bitmap
      btrfs: Remove fs_info argument from add_free_space_extent
      btrfs: Remove fs_info argument from remove_free_space_extent
      btrfs: Remove fs_info argument from __remove_from_free_space_tree
      btrfs: Remove fs_info argument from remove_from_free_space_tree
      btrfs: Remove fs_info argument from add_to_free_space_tree
      btrfs: Remove fs_info argument from populate_free_space_tree
      btrfs: Unexport and rename btrfs_invalidate_inodes
      btrfs: Remove stale comment about select_delayed_ref
      btrfs: Remove fs_info argument from alloc_reserved_tree_block
      btrfs: Simplify alloc_reserved_tree_block interface
      btrfs: Pass btrfs_delayed_extent_op to alloc_reserved_tree_block
      btrfs: Streamline shared ref check in alloc_reserved_tree_block
      btrfs: Factor out read portion of btrfs_get_blocks_direct
      btrfs: Factor out write portion of btrfs_get_blocks_direct

Omar Sandoval (16):
      Btrfs: update stale comments referencing vmtruncate()
      Btrfs: fix error handling in btrfs_truncate_inode_items()
      Btrfs: don't BUG_ON() in btrfs_truncate_inode_items()
      Btrfs: stop creating orphan items for truncate
      Btrfs: get rid of BTRFS_INODE_HAS_ORPHAN_ITEM
      Btrfs: delete dead code in btrfs_orphan_commit_root()
      Btrfs: don't return ino to ino cache if inode item removal fails
      Btrfs: refactor btrfs_evict_inode() reserve refill dance
      Btrfs: fix ENOSPC caused by orphan items reservations
      Btrfs: get rid of unused orphan infrastructure
      Btrfs: renumber BTRFS_INODE_ runtime flags and switch to enums
      Btrfs: reserve space for O_TMPFILE orphan item deletion
      Btrfs: allow empty subvol= again
      Btrfs: fix clone vs chattr NODATASUM race
      Btrfs: fix memory and mount leak in btrfs_ioctl_rm_dev_v2()
      Btrfs: clean up error handling in btrfs_truncate()

Qu Wenruo (15):
      btrfs: print-tree: Add eb locking status output for debug build
      btrfs: trace: Remove unnecessary fs_info parameter for btrfs__reserve_extent event class
      btrfs: trace: Add trace points for unused block groups
      btrfs: trace: Allow trace_qgroup_update_counters() to record old rfer/excl value
      btrfs: qgroup: Allow trace_btrfs_qgroup_account_extent() to record its transid
      btrfs: Move btrfs_check_super_valid() to avoid forward declaration
      btrfs: Refactor btrfs_check_super_valid
      btrfs: Do super block verification before writing it to disk
      btrfs: qgroup: Search commit root for rescan to avoid missing extent
      btrfs: qgroup: Finish rescan when hit the last leaf of extent tree
      btrfs: compression: Add linux/sizes.h for compression.h
      btrfs: lzo: document the compressed data format
      btrfs: lzo: Add header length check to avoid potential out-of-bounds access
      btrfs: lzo: Harden inline lzo compressed extent decompression
      btrfs: qgroup: show more meaningful qgroup_rescan_init error message

Robbie Ko (2):
      btrfs: incremental send, move allocation until it's needed in orphan_dir_info
      btrfs: incremental send, improve rmdir performance for large directory

Su Yue (3):
      btrfs: rename btrfs_get_block_group_info and make it static
      btrfs: return error value if create_io_em failed in cow_file_range
      btrfs: return ENOMEM if path allocation fails in btrfs_cross_ref_exist

Timofey Titovets (3):
      Btrfs: split btrfs_extent_same
      Btrfs: dedupe_file_range ioctl: remove 16MiB restriction
      Btrfs: reuse cmp workspace in EXTENT_SAME ioctl

Tomohiro Misono (4):
      btrfs: sysfs: Use enum/define value for feature array definitions
      btrfs: Add unprivileged ioctl which returns subvolume information
      btrfs: Add unprivileged ioctl which returns subvolume's ROOT_REF
      btrfs: Add unprivileged version of ino_lookup ioctl

 fs/btrfs/btrfs_inode.h                 |   22 +-
 fs/btrfs/compression.c                 |    7 +-
 fs/btrfs/compression.h                 |    2 +
 fs/btrfs/ctree.c                       |  123 +--
 fs/btrfs/ctree.h                       |   76 +-
 fs/btrfs/delayed-inode.c               |    9 +-
 fs/btrfs/delayed-ref.c                 |  275 +++----
 fs/btrfs/delayed-ref.h                 |    5 +-
 fs/btrfs/dev-replace.c                 |  150 +++-
 fs/btrfs/disk-io.c                     |  391 +++++----
 fs/btrfs/extent-tree.c                 |  253 +++---
 fs/btrfs/extent_io.c                   |   62 +-
 fs/btrfs/extent_io.h                   |   20 +-
 fs/btrfs/extent_map.c                  |    6 +-
 fs/btrfs/extent_map.h                  |    3 +-
 fs/btrfs/free-space-cache.c            |    6 +-
 fs/btrfs/free-space-tree.c             |  192 +++--
 fs/btrfs/free-space-tree.h             |    8 -
 fs/btrfs/inode.c                       | 1371 ++++++++++++++++----------------
 fs/btrfs/ioctl.c                       | 1210 ++++++++++++++++++----------
 fs/btrfs/locking.c                     |   34 +-
 fs/btrfs/lzo.c                         |   76 +-
 fs/btrfs/ordered-data.c                |   14 +-
 fs/btrfs/print-tree.c                  |   21 +
 fs/btrfs/qgroup.c                      |   69 +-
 fs/btrfs/raid56.c                      |   38 +-
 fs/btrfs/relocation.c                  |    8 +-
 fs/btrfs/scrub.c                       |    1 +
 fs/btrfs/send.c                        |   46 +-
 fs/btrfs/super.c                       |    7 +-
 fs/btrfs/sysfs.c                       |   52 +-
 fs/btrfs/sysfs.h                       |    4 +-
 fs/btrfs/tests/btrfs-tests.c           |    4 +-
 fs/btrfs/tests/btrfs-tests.h           |    6 +-
 fs/btrfs/tests/extent-buffer-tests.c   |   56 +-
 fs/btrfs/tests/extent-io-tests.c       |   75 +-
 fs/btrfs/tests/extent-map-tests.c      |   90 ++-
 fs/btrfs/tests/free-space-tests.c      |  177 +++--
 fs/btrfs/tests/free-space-tree-tests.c |  129 +--
 fs/btrfs/tests/inode-tests.c           |  312 ++++----
 fs/btrfs/tests/qgroup-tests.c          |  100 +--
 fs/btrfs/transaction.c                 |   15 +-
 fs/btrfs/transaction.h                 |    1 -
 fs/btrfs/tree-log.c                    |   28 +-
 fs/btrfs/uuid-tree.c                   |   10 +-
 fs/btrfs/volumes.c                     |  506 ++++++------
 fs/btrfs/volumes.h                     |   24 +-
 include/trace/events/btrfs.h           |  323 ++++----
 include/uapi/linux/btrfs.h             |   97 +++
 49 files changed, 3579 insertions(+), 2935 deletions(-)