2020 NYE Patchbomb Guide!

2020 NYE Patchbomb Guide!

[Darrick J. Wong]

Hi folks,

It's still the last day of 2020 in the US/Pacific timezone, which means
it's time for me to patchbomb every new feature that's been sitting
around in my development trees!

Once a year I like to dump everything into the public archives to
increase XFS' bus factor.  However, there are 660 patches and 46 cover
letters across my three development trees, so this year I'm only sending
the guide and the cover letters.

Just like last year, this guide lists each feature branch that's merged
into the development tree, along with some notes to help readers figure
out the real dependencies between branches.

If you want to try out this code, you /really/ should pull from
kernel.org:

https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git

==============

The kernel tree contains the following branches:

 pwork-parallelism	("xfs: increase pwork parallelism")

Single patch to increase parallelism of background threads to handle
high-iops storage.  At this point in the branch the only piece to need
this is quotacheck, and I haven't been able to show that parallel
quotacheck is any faster due to the bottlenecks shifting to memory
reclaim, but deferred inode inactivation will want this.

 scrub-fixes		("xfs: more scrub fixes")

The usual pile of bug fixes and verification strengthening for scrub.

 reflink-speedups	("xfs: fix reflink inefficiencies")

This independent series cleans up some warts with bunmapi when reflink
is enabled, since we've now restructured various shortcomings in the
defer ops code and the log that made some dirty hacks unnecessary.

 repair-reap-fixes	("xfs: fix online repair block reaping")
 repair-bitmap-rework	("xfs: rework online repair incore bitmap")
 repair-prep-for-bulk-loading ("xfs: prepare repair for bulk loading")
 scrub-rtsummary	("xfs: online scrubbing of realtime summary files")
 repair-ag-btrees	("xfs: online repair of AG btrees")
 repair-inodes		("xfs: online repair of inodes and extent maps")
 repair-quota		("xfs: online repair of quota and counters")
 corruption-health-reports ("xfs: report corruption to the health trackers")

Here's the first part of online filesystem repair.  Since last year,
we've landed the btree bulk loading code, which shortened this story
arc by one.  However, I've also added the ability to check the realtime
summary metadata, and strengthened the rebuilder code to check the
sanity of the records it collects during a rebuild process.

 reclaim-space-harder	("xfs: try harder to reclaim space when we run out")
 eofblocks-consolidation ("xfs: consolidate posteof and cowblocks cleanup")
 deferred-inactivation	("xfs: deferred inode inactivation")

I think I'll send these three branches for the next kernel.  The third
branch implements deferred inode inactivation, which is to say that we
move all file deletion activities to background threads.  This makes
unlink calls return to userspace faster and should speed up mass
deletions since we now process inodes in disk order instead of deletion
order.  The first series makes ENOSPC space reclamation push the
filesystem harder so that it will behave the same after we push inode
inactivation to a background workqueue.  The second series combines
garbage collection of speculative post-EOF blocks and COW extents into a
single piece of code so that we can free up one radix tree tag bit.

(None of this code require online repair...)

 indirect-health-reporting ("xfs: indirect health reporting")

This series improves the metadata health reporting system so that it can
remember health problems for principal filesystem objects whose incore
representations had to be reclaimed.  In other words, the per-AG state
can remember the fact that an inode in that AG had problems, even if we
have to reclaim the xfs_inode for that broken file.

 repair-hard-problems	("xfs: online repair hard problems")

This branch contains harder repair problems, namely the ones that
require full filesystem scans.  These scans depend upon being able to
shut down background operations temporarily, which is we need deferred
inode inactivation (and part 1 of online repair) to solve these
problems.

 realtime-bmap-intents	("xfs: widen BUI formats to support rt")
 expand-bmap-intent-usage ("xfs: support attrfork and unwritten BUIs")
 atomic-file-updates	("xfs: atomic file updates")

A key part of the third installment of online filesystem repair will be
the ability to switch the contents of xattrs, directories, rt metadata,
and symbolic links atomically.  For this, I developed a new high-level
log operation to swap the extents of any two file forks and to resume
the swap even if the system goes down during the operation.  I also
wired this up to a userspace ioctl so that user programs can commit file
content updates atomically.

 repair-rtsummary	("xfs: online repair of realtime summaries")
 repair-xattrs		("xfs: online repair of extended attributes")
 repair-dirs		("xfs: online repair of directories")

This third piece of online filesystem repair builds upon the atomic
extent swap in the previous batch.  It enables us to rebuild file-based
metadata.

 inode-refactor		("xfs: hoist inode operations to libxfs")
 metadir		("xfs: metadata inode directories")

The first branch in this series hoists all the inode allocation and
freeing code to libxfs so that the second series can create arbitrary
directory trees for metadata.  This will also allow us to refactor all
of the similar but not identical code in userspace so that file
attributes can be inherited in a consistent manner when constructing a
filesystem from a protofile.

 btree-ifork-records	("xfs: refactor btrees to support records in inode root")
 btree-dynamic-depth	("xfs: support dynamic btree cursor height")

These three branches refactor the generic btree code to support using
inode root areas for btree records, which will be needed for realtime
rmap and reflink.

 refactor-rt-locking	("xfs: refactor realtime meta inode locking")
 reserve-rt-metadata-space ("xfs: enable in-core block reservation for rt metadata")

Once I started running fstests on rmap/reflink-enabled rt filesystems, I
noticed that I was running into weird deadlocks and space allocation
failure issues that happened with the original rmap and reflink
implementations.  The first series refactors locking of rt metadata so
that we can do it in a systematic way, and the second series extends the
per-AG allocation code to work on realtime devices too.

 realtime-extfree-intents ("xfs: widen EFI format to support rt")
 realtime-rmap		("xfs: realtime reverse-mapping support")

This series uses the metadata directory tree to add reverse mapping
abilities to the realtime device, and using that rmap data to
reconstruct realtime free space metadata.  It depends on the six
branches above.

 noalloc-ags		("xfs: noalloc allocation groups")

A quick branch I whipped up while exploring online filesystem shrink,
which gives us the ability to prevent allocations in an AG.  It doesn't
depend on any of the branches above it; this is just where it landed in
my development tree.

 realtime-reflink	("xfs: reflink on the realtime device")
 realtime-reflink-extsize ("xfs: reflink with large realtime extents")

This last series uses the metadata directory tree to create a new
refcount btree for the realtime device.  With that, we can support
reflink on realtime devices.  This requires every branch from
inode-refactor to reserve-rt-metadata-space.

==============

On the xfsprogs side of things, here are some significant branches that
I would like to highlight.  The branch groups do not depend on each other.

 xfs_db-directory-navigation ("xfs_db: add minimal directory navigation")

This branch adds directory navigation and directory listing abilities to
xfs_db.

 needsrepair		("xfs: add the ability to flag a fs for repair")
 fs-upgrades		("xfs_admin: support upgrading v5 filesystems")

These two branches enable sysadmins to add (some) new features to
existing V5 filesystems.

 repair-rebuild-forks	("xfs_repair: rebuild inode fork mappings")

This branch teaches xfs_repair to rebuild inode fork mappings from rmap
data.

 packaging-cleanups	("xfsprogs: packaging cleanups")
 bmap-utils		("xfsprogs: file write utility refactoring")

A couple of cleanups for userspace.

 scrub-iscan-rebalance	("xfs_scrub: improve balancing of threads for inode scan")

This is a major rework of inode scans in xfs_scrub.  The first branch
parallelises inode scans at a finer granularity -- now each worker
thread deals with an inobt record, and not an entire AG.  This reduces
the amount of pending scrub work that can stall behind a single large
fragmented file.

 xfs-scrub-fixes	("xfs_scrub: second fixes series")
 scrub-repair-data-deps	("xfs_scrub: track data dependencies for repairs")

These last two restructure how we track metadata repair activities so
that it is done on a per-fs principal (AGs, inodes) basis instead of
tracking each repair individually.  What this means is that we now
preserve ordering dependencies between repair types.  That makes us
smart enough not to try to repair a directory if the data bmbt is still
broken.

==============

Finally, here some significant branches awaiting review for fstests:

 fuzz-baseline		("xfstests: establish baseline for fuzz tests")
 fuzzer-improvements	("xfstests: improve xfs fuzzing")
 more-fuzz-testing	("xfstests: strengthen fuzz testing")

In these three branches, I rework the XFS metadata fuzz testing
infrastructure to provide /some/ sort of a baseline golden output, and
to be a bit more systematic about how it reports where it is in a fuzz
process.  I also rework the fuzz repair strategies so that they more
closely follow the known reaction cases (no repair at all; online repair
only; offline repair only; and online repair followed by offline repair
if needed).

 dmerror-on-rt-devices	("common/dm*: support external log and rt devices")

This makes it so that we can use dmerror and dmflakey for external log
and realtime devices.

With that, I hope you all have a happy new year!  See you in 2021!

--D

[PATCHSET 0/1] xfs: increase pwork parallelism

[Darrick J. Wong]

Hi all,

Increase the parallelization factor of the pwork infrastructure so that
we can scale to as many CPUs and AGs as there are in the filesystem.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=pwork-parallelism
---
 fs/xfs/xfs_buf.c   |   34 ++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_buf.h   |    1 +
 fs/xfs/xfs_iwalk.c |    2 +-
 fs/xfs/xfs_mount.c |   39 +++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_mount.h |    1 +
 fs/xfs/xfs_pwork.c |   17 +++++------------
 fs/xfs/xfs_pwork.h |    2 +-
 7 files changed, 82 insertions(+), 14 deletions(-)

[PATCHSET 00/11] xfs: more scrub fixes

[Darrick J. Wong]

Hi all,

Fix a deficiency in the scrubbers where we don't quite properly detect
gaps in records.  Also, enhance the fscounters scrubber to check the
free rt extent count, look for oversized rmaps when checking the bmap,
and verify the non-shared status of extents not explicitly covered by
refcount records.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-fixes

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-fixes
---
 fs/xfs/libxfs/xfs_alloc.c      |   23 +++++++++---
 fs/xfs/libxfs/xfs_btree.c      |   79 +++++++++++++++++++++++++++++++++++++---
 fs/xfs/libxfs/xfs_btree.h      |    6 +++
 fs/xfs/libxfs/xfs_ialloc.c     |    4 +-
 fs/xfs/libxfs/xfs_inode_fork.c |    2 +
 fs/xfs/libxfs/xfs_refcount.c   |   15 +++++++-
 fs/xfs/libxfs/xfs_rmap.c       |   15 +++++++-
 fs/xfs/scrub/agheader.c        |   33 +++++------------
 fs/xfs/scrub/bmap.c            |   74 ++++++++++++++++++++++++++++++++++++-
 fs/xfs/scrub/btree.c           |   49 ++++++++++++++++++++++++-
 fs/xfs/scrub/common.c          |   23 +++++-------
 fs/xfs/scrub/common.h          |    5 +--
 fs/xfs/scrub/fscounters.c      |   50 +++++++++++++++++++++++++
 fs/xfs/scrub/health.c          |    3 +-
 fs/xfs/scrub/quota.c           |    6 ++-
 fs/xfs/scrub/refcount.c        |   80 ++++++++++++++++++++++++++++++++++++++--
 fs/xfs/scrub/scrub.c           |    2 +
 fs/xfs/scrub/scrub.h           |    1 +
 18 files changed, 402 insertions(+), 68 deletions(-)

[PATCHSET 0/5] xfs: fix reflink inefficiencies

[Darrick J. Wong]

Hi all,

As Dave Chinner has complained about on IRC, there are a couple of
things about reflink that are very inefficient.  First of all, we
limited the size of all bunmapi operations to avoid flooding the log
with defer ops in the worst case, but recent changes to the defer ops
code have solved that problem, so get rid of the bunmapi length clamp.

Second, the log reservations for reflink operations are far far larger
than they need to be.  Shrink them to exactly what we need to handle
each deferred RUI and CUI log item, and no more.  Also reduce logcount
because we don't need 8 rolls per operation.  Introduce a transaction
reservation compatibility layer to avoid changing the minimum log size
calculations.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=reflink-speedups

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=reflink-speedups
---
 fs/xfs/libxfs/xfs_bmap.c       |   22 -----
 fs/xfs/libxfs/xfs_log_rlimit.c |   17 +++-
 fs/xfs/libxfs/xfs_refcount.c   |   14 ++-
 fs/xfs/libxfs/xfs_refcount.h   |    8 --
 fs/xfs/libxfs/xfs_trans_resv.c |  168 +++++++++++++++++++++++++++++++++++-----
 fs/xfs/libxfs/xfs_trans_resv.h |    8 +-
 fs/xfs/xfs_reflink.c           |    7 +-
 fs/xfs/xfs_trace.h             |   12 ++-
 8 files changed, 194 insertions(+), 62 deletions(-)

[PATCHSET v2 0/2] xfs: fix online repair block reaping

[Darrick J. Wong]

Hi all,

These patches fix a few problems that I noticed in the code that deals
with old btree blocks after a successful repair. First, we clarify how
the reaping function works w.r.t. bitmap lifetimes.  Next we fix a bug
where we could incorrectly invalidate old btree blocks if they were
crosslinked.  Finally, we convert the reap function to use EFIs so that
we can delete blocks without overloading a transaction.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-reap-fixes
---
 fs/xfs/scrub/repair.c |  115 ++++++++++++++++++++++++++++---------------------
 fs/xfs/scrub/repair.h |    2 -
 2 files changed, 67 insertions(+), 50 deletions(-)

[PATCHSET v2 0/3] xfs: rework online repair incore bitmap

[Darrick J. Wong]

Hi all,

In this series, we make some changes to the incore bitmap code: First,
we shorten the prefix to 'xbitmap'.  Then, we rework some utility
functions for later use by online repair and clarify how the walk
functions are supposed to be used.

Finally, we use all these new pieces to convert the incore bitmap to use
an interval tree instead of linked lists.  This lifts the limitation
that callers had to be careful not to set a range that was already set;
and gets us ready for the btree rebuilder functions needing to be able
to set bits in a bitmap and generate maximal contiguous extents for the
set ranges.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-bitmap-rework
---
 fs/xfs/scrub/agheader_repair.c |   77 +++++----
 fs/xfs/scrub/bitmap.c          |  348 +++++++++++++++++++++++-----------------
 fs/xfs/scrub/bitmap.h          |   35 ++--
 fs/xfs/scrub/repair.c          |   60 ++++---
 4 files changed, 297 insertions(+), 223 deletions(-)

[PATCHSET v2 0/3] xfs: prepare repair for bulk loading

[Darrick J. Wong]

Hi all,

Add some new helper functions (and debug knobs) to online repair to
prepare us to stage new btrees.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-prep-for-bulk-loading
---
 fs/xfs/scrub/agheader_repair.c |    1 
 fs/xfs/scrub/common.c          |    1 
 fs/xfs/scrub/repair.c          |  395 ++++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/repair.h          |   58 ++++++
 fs/xfs/scrub/scrub.c           |    2 
 fs/xfs/scrub/trace.h           |   36 ++++
 fs/xfs/xfs_extfree_item.c      |    2 
 fs/xfs/xfs_globals.c           |   12 +
 fs/xfs/xfs_sysctl.h            |    2 
 fs/xfs/xfs_sysfs.c             |   54 +++++
 10 files changed, 561 insertions(+), 2 deletions(-)

[PATCHSET 0/3] xfs: online scrubbing of realtime summary files

[Darrick J. Wong]

Hi all,

This patchset implements an online checker for the realtime summary
file.  We begin with the xfile code from the repair series, which we use
here to build an in-memory copy of the summary information from the
ondisk realtime bitmap that can be paged out to disk.

Next, we move the rtsummary scrubber stub function into a separate file.

The final patch in the series implements the logic to build the
rtsummary file and compare it to the existing contents.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-rtsummary
---
 fs/xfs/Kconfig           |    1 
 fs/xfs/Makefile          |    8 -
 fs/xfs/scrub/array.c     |  665 ++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/array.h     |   49 +++
 fs/xfs/scrub/common.h    |   10 +
 fs/xfs/scrub/rtbitmap.c  |   41 ---
 fs/xfs/scrub/rtsummary.c |  327 +++++++++++++++++++++++
 fs/xfs/scrub/scrub.c     |    9 -
 fs/xfs/scrub/scrub.h     |    7 
 fs/xfs/scrub/trace.c     |    1 
 fs/xfs/scrub/trace.h     |  118 ++++++++
 fs/xfs/scrub/xfile.c     |  271 +++++++++++++++++++
 fs/xfs/scrub/xfile.h     |   23 ++
 13 files changed, 1486 insertions(+), 44 deletions(-)
 create mode 100644 fs/xfs/scrub/array.c
 create mode 100644 fs/xfs/scrub/array.h
 create mode 100644 fs/xfs/scrub/rtsummary.c
 create mode 100644 fs/xfs/scrub/xfile.c
 create mode 100644 fs/xfs/scrub/xfile.h

[PATCHSET v22 0/4] xfs: online repair of AG btrees

[Darrick J. Wong]

Hi all,

This is the first part of the twenty-second revision of a patchset that
adds to XFS kernel support for online metadata scrubbing and repair.
There aren't any on-disk format changes.

New for this version is a rebase against 5.5-rc4, bulk loading of
btrees, integration with the health reporting subsystem, and the
explicit revalidation of all metadata structures that were rebuilt.

First, create a new data structure that provides an abstraction of a big
memory array by using linked lists.  This is where we store records for
btree reconstruction.  This first implementation is memory inefficient
and consumes a /lot/ of kernel memory, but lays the groundwork for a
later patch in the set to convert the implementation to use a (memfd)
swap file, which enables us to use pageable memory without pounding the
slab cache.

The three patches after that implement reconstruction of the free space
btrees, inode btrees, and reference count btree.  The reverse mapping
btree requires considerably more thought and will be covered later.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-ag-btrees
---
 fs/xfs/Makefile                  |    3 
 fs/xfs/libxfs/xfs_ag_resv.c      |    2 
 fs/xfs/libxfs/xfs_ialloc.c       |   20 +
 fs/xfs/libxfs/xfs_ialloc.h       |    1 
 fs/xfs/libxfs/xfs_ialloc_btree.c |    2 
 fs/xfs/libxfs/xfs_ialloc_btree.h |    2 
 fs/xfs/libxfs/xfs_types.c        |   20 +
 fs/xfs/libxfs/xfs_types.h        |    9 
 fs/xfs/scrub/alloc_repair.c      |  761 +++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/common.c            |    9 
 fs/xfs/scrub/health.c            |   10 
 fs/xfs/scrub/ialloc_repair.c     |  837 ++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/refcount_repair.c   |  653 ++++++++++++++++++++++++++++++
 fs/xfs/scrub/repair.c            |   66 +++
 fs/xfs/scrub/repair.h            |   32 +
 fs/xfs/scrub/scrub.c             |   19 +
 fs/xfs/scrub/scrub.h             |    9 
 fs/xfs/scrub/trace.h             |  103 +++--
 fs/xfs/xfs_extent_busy.c         |   13 +
 fs/xfs/xfs_extent_busy.h         |    2 
 20 files changed, 2517 insertions(+), 56 deletions(-)
 create mode 100644 fs/xfs/scrub/alloc_repair.c
 create mode 100644 fs/xfs/scrub/ialloc_repair.c
 create mode 100644 fs/xfs/scrub/refcount_repair.c

[PATCHSET v22 0/4] xfs: online repair of inodes and extent maps

[Darrick J. Wong]

Hi all,

In the second part of the twenty-second revision of the online repair
patchset, we implement repairing of inode records, inode forks, and
inode block maps.  We also implement a meager symbolic link repair
ability.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-inodes

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-inodes
---
 fs/xfs/Makefile                   |    3 
 fs/xfs/libxfs/xfs_attr_leaf.c     |   32 +
 fs/xfs/libxfs/xfs_attr_leaf.h     |    2 
 fs/xfs/libxfs/xfs_bmap.c          |   22 +
 fs/xfs/libxfs/xfs_bmap.h          |    2 
 fs/xfs/libxfs/xfs_bmap_btree.c    |  117 +++-
 fs/xfs/libxfs/xfs_bmap_btree.h    |    8 
 fs/xfs/libxfs/xfs_btree_staging.c |   11 
 fs/xfs/libxfs/xfs_btree_staging.h |    8 
 fs/xfs/libxfs/xfs_format.h        |    3 
 fs/xfs/libxfs/xfs_iext_tree.c     |   23 +
 fs/xfs/libxfs/xfs_inode_fork.c    |    1 
 fs/xfs/libxfs/xfs_inode_fork.h    |    3 
 fs/xfs/scrub/bmap.c               |   22 +
 fs/xfs/scrub/bmap_repair.c        |  637 ++++++++++++++++++++++
 fs/xfs/scrub/inode_repair.c       | 1053 +++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/refcount_repair.c    |    6 
 fs/xfs/scrub/repair.h             |   24 +
 fs/xfs/scrub/scrub.c              |    8 
 fs/xfs/scrub/symlink.c            |    5 
 fs/xfs/scrub/symlink_repair.c     |  245 +++++++++
 fs/xfs/scrub/trace.h              |   34 +
 fs/xfs/xfs_symlink.c              |  147 +++--
 fs/xfs/xfs_symlink.h              |    3 
 fs/xfs/xfs_trans.c                |   54 ++
 fs/xfs/xfs_trans.h                |    2 
 26 files changed, 2347 insertions(+), 128 deletions(-)
 create mode 100644 fs/xfs/scrub/bmap_repair.c
 create mode 100644 fs/xfs/scrub/inode_repair.c
 create mode 100644 fs/xfs/scrub/symlink_repair.c

[PATCHSET v22 0/6] xfs: online repair of quota and counters

[Darrick J. Wong]

Hi all,

This series starts by implementing quick repairs for obvious problems
with quota records.  Next, we implement a new scrubber to check all
quota counters in the system.  This is done by walking all inodes (just
like mount time quotacheck) but with the added twist that the online
quotacheck code hooks the regular quota update mechanism to stay abreast
of changes to quota counters, thereby ensuring that the quota counters
are always up to date.  The series ends by adding the ability to commit
the new counters to the dquots.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quota

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-quota

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-quota
---
 fs/xfs/Kconfig                   |    1 
 fs/xfs/Makefile                  |   11 
 fs/xfs/libxfs/xfs_bmap.c         |   41 ++
 fs/xfs/libxfs/xfs_bmap.h         |    3 
 fs/xfs/libxfs/xfs_fs.h           |    3 
 fs/xfs/scrub/bmap_repair.c       |   16 +
 fs/xfs/scrub/common.c            |   11 
 fs/xfs/scrub/common.h            |   16 +
 fs/xfs/scrub/quota.c             |   12 -
 fs/xfs/scrub/quota.h             |   11 
 fs/xfs/scrub/quota_repair.c      |  401 ++++++++++++++++++
 fs/xfs/scrub/quotacheck.c        |  838 ++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/quotacheck.h        |   73 +++
 fs/xfs/scrub/quotacheck_repair.c |  247 +++++++++++
 fs/xfs/scrub/repair.c            |   80 ++++
 fs/xfs/scrub/repair.h            |   12 +
 fs/xfs/scrub/scrub.c             |   20 +
 fs/xfs/scrub/scrub.h             |   12 +
 fs/xfs/scrub/trace.c             |    1 
 fs/xfs/scrub/trace.h             |   27 +
 fs/xfs/xfs_inode.c               |   26 -
 fs/xfs/xfs_iwalk.c               |  118 +++++
 fs/xfs/xfs_iwalk.h               |    3 
 fs/xfs/xfs_mount.c               |   43 ++
 fs/xfs/xfs_mount.h               |   10 
 fs/xfs/xfs_qm.c                  |   18 -
 fs/xfs/xfs_qm.h                  |   15 +
 fs/xfs/xfs_quota.h               |   19 +
 fs/xfs/xfs_trans_dquot.c         |   72 +++
 29 files changed, 2100 insertions(+), 60 deletions(-)
 create mode 100644 fs/xfs/scrub/quota.h
 create mode 100644 fs/xfs/scrub/quota_repair.c
 create mode 100644 fs/xfs/scrub/quotacheck.c
 create mode 100644 fs/xfs/scrub/quotacheck.h
 create mode 100644 fs/xfs/scrub/quotacheck_repair.c

[PATCHSET 00/10] xfs: report corruption to the health trackers

[Darrick J. Wong]

Hi all,

In a previous series we performed a lot of cleanups to the metadata
corruption detection code in XFS.  Now add the ability to report these
problems to the health tracking system so that administrators can gather
reports on live filesystems, and (in the future) to enable more targeted
scanning of metadata.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=corruption-health-reports

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=corruption-health-reports

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=corruption-health-reports
---
 fs/xfs/libxfs/xfs_alloc.c       |  103 ++++++++++++++++++++----
 fs/xfs/libxfs/xfs_attr_leaf.c   |    5 +
 fs/xfs/libxfs/xfs_attr_remote.c |   25 +++---
 fs/xfs/libxfs/xfs_bmap.c        |  115 ++++++++++++++++++++++++--
 fs/xfs/libxfs/xfs_btree.c       |   30 ++++++-
 fs/xfs/libxfs/xfs_da_btree.c    |   35 +++++++-
 fs/xfs/libxfs/xfs_dir2.c        |    5 +
 fs/xfs/libxfs/xfs_dir2_block.c  |    2 
 fs/xfs/libxfs/xfs_dir2_data.c   |    3 +
 fs/xfs/libxfs/xfs_dir2_leaf.c   |    3 +
 fs/xfs/libxfs/xfs_dir2_node.c   |    7 ++
 fs/xfs/libxfs/xfs_health.h      |   35 +++++++-
 fs/xfs/libxfs/xfs_ialloc.c      |   56 +++++++++++--
 fs/xfs/libxfs/xfs_inode_buf.c   |    4 +
 fs/xfs/libxfs/xfs_inode_fork.c  |    8 ++
 fs/xfs/libxfs/xfs_refcount.c    |   36 ++++++++
 fs/xfs/libxfs/xfs_rmap.c        |   86 +++++++++++++++++++-
 fs/xfs/libxfs/xfs_rmap.h        |    2 
 fs/xfs/libxfs/xfs_rtbitmap.c    |    9 ++
 fs/xfs/libxfs/xfs_sb.c          |    2 
 fs/xfs/scrub/health.c           |   20 +++--
 fs/xfs/scrub/refcount_repair.c  |    4 +
 fs/xfs/scrub/rmap.c             |    2 
 fs/xfs/xfs_attr_inactive.c      |    4 +
 fs/xfs/xfs_attr_list.c          |    7 +-
 fs/xfs/xfs_discard.c            |    2 
 fs/xfs/xfs_dquot.c              |   28 ++++++
 fs/xfs/xfs_health.c             |  170 +++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_icache.c             |    5 +
 fs/xfs/xfs_inode.c              |   11 +++
 fs/xfs/xfs_iomap.c              |   15 +++
 fs/xfs/xfs_iops.c               |    5 +
 fs/xfs/xfs_iwalk.c              |    4 +
 fs/xfs/xfs_mount.c              |    5 +
 fs/xfs/xfs_qm.c                 |    9 ++
 fs/xfs/xfs_rtalloc.c            |    5 +
 fs/xfs/xfs_symlink.c            |    4 +
 37 files changed, 784 insertions(+), 87 deletions(-)

[PATCHSET v2 0/6] xfs: try harder to reclaim space when we run out

[Darrick J. Wong]

Hi all,

Historically, when users ran out of space or quota when trying to write
to the filesystem, XFS didn't try very hard to reclaim space that it
might have been hanging onto for the purpose of speeding up front-end
filesystem operations (appending writes, cow staging).  The upcoming
deferred inactivation series will greatly increase the amount of
allocated space that isn't actively being used to store user data.

Therefore, try to reduce the circumstances where we return EDQUOT or
ENOSPC to userspace by teaching the write paths to try to clear space
and retry the operation one time before giving up.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=reclaim-space-harder
---
 fs/xfs/xfs_bmap_util.c   |   29 ++++++++++
 fs/xfs/xfs_file.c        |   24 +++-----
 fs/xfs/xfs_icache.c      |  132 +++++++++++++++++++++++++++++++++++-----------
 fs/xfs/xfs_icache.h      |   14 ++---
 fs/xfs/xfs_inode.c       |   29 +++++++++-
 fs/xfs/xfs_ioctl.c       |    2 +
 fs/xfs/xfs_iomap.c       |   32 +++++++++++
 fs/xfs/xfs_qm_syscalls.c |    3 -
 fs/xfs/xfs_reflink.c     |   63 +++++++++++++++++++++-
 fs/xfs/xfs_trace.c       |    1 
 fs/xfs/xfs_trace.h       |   41 ++++++++++++++
 11 files changed, 306 insertions(+), 64 deletions(-)

[PATCHSET 0/6] xfs: consolidate posteof and cowblocks cleanup

[Darrick J. Wong]

Hi all,

Currently, we treat the garbage collection of post-EOF preallocations
and copy-on-write preallocations as totally separate tasks -- different
incore inode tags, different workqueues, etc.  This is wasteful of radix
tree tags and workqueue resources since we effectively have parallel
code paths to do the same thing.

Therefore, consolidate both functions under one radix tree bit and one
workqueue function that scans an inode for both things at the same time.
At the end of the series we make the scanning per-AG instead of per-fs
so that the scanning can run in parallel.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=eofblocks-consolidation

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=eofblocks-consolidation
---
 fs/xfs/scrub/common.c  |    4 -
 fs/xfs/xfs_bmap_util.c |  109 ++++++++++------------
 fs/xfs/xfs_globals.c   |    7 +
 fs/xfs/xfs_icache.c    |  237 +++++++++++++++++++++++-------------------------
 fs/xfs/xfs_icache.h    |   13 +--
 fs/xfs/xfs_inode.c     |   36 +++++++
 fs/xfs/xfs_inode.h     |    2 
 fs/xfs/xfs_linux.h     |    3 -
 fs/xfs/xfs_mount.c     |    5 +
 fs/xfs/xfs_mount.h     |    9 +-
 fs/xfs/xfs_super.c     |   43 ++++++---
 fs/xfs/xfs_sysctl.c    |   15 +--
 fs/xfs/xfs_sysctl.h    |    3 -
 fs/xfs/xfs_trace.h     |    6 -
 14 files changed, 256 insertions(+), 236 deletions(-)

[PATCHSET v2 0/7] xfs: deferred inode inactivation

[Darrick J. Wong]

Hi all,

This patch series implements deferred inode inactivation.  Inactivation
is the process of updating all on-disk metadata when a file is deleted
-- freeing the data/attr/COW fork extent allocations, removing the inode
from the unlinked hash, marking the inode record itself free, and
updating the inode btrees so that they show the inode as not being in
use.

Currently, all this inactivation is performed during in-core inode
reclaim, which creates two big headaches: first, this makes direct
memory reclamation /really/ slow, and second, it prohibits us from
partially freezing the filesystem for online fsck activity because scrub
can hit direct memory reclaim.  It's ok for scrub to fail with ENOMEM,
but it's not ok for scrub to deadlock memory reclaim. :)

The implementation will be familiar to those who have studied how XFS
scans for reclaimable in-core inodes -- we create a couple more inode
state flags to mark an inode as needing inactivation and being in the
middle of inactivation.  When inodes need inactivation, we set iflags,
set the RECLAIM radix tree tag, update a count of how many resources
will be freed by the pending inactivations, and schedule a deferred work
item.  The deferred work item scans the inode radix tree for inodes to
inactivate, and does all the on-disk metadata updates.  Once the inode
has been inactivated, it is left in the reclaim state and the background
reclaim worker (or direct reclaim) will get to it eventually.

Patch 1-2 refactor some of the inactivation predicates.

Patches 3-4 implement the count of blocks/quota that can be freed by
running inactivation; this is necessary to preserve the behavior where
you rm a file and the fs counters update immediately.

Patches 5-6 refactor more inode reclaim code so that we can reuse some
of it for inactivation.

Patch 8 delivers the core of the inactivation changes by altering the
inode lifetime state machine to include the new inode flags and
background workers.

Patches 9-10 makes it so that if an allocation attempt hits ENOSPC it
will force inactivation to free resources and try again.

Patch 11 converts the per-fs inactivation scanner to be tracked on a
per-AG basis so that we can be more targeted in our inactivation.

Patches 12-14 teach the per-AG sick status to remember if we inactivate
inodes that themselves had unfixed sick flags set, and for scrub to
clear all those flags if it finds that the filesystem is clean.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=deferred-inactivation
---
 fs/xfs/scrub/common.c     |    2 
 fs/xfs/scrub/quotacheck.c |    7 +
 fs/xfs/xfs_bmap_util.c    |   38 +++
 fs/xfs/xfs_fsops.c        |    9 +
 fs/xfs/xfs_icache.c       |  555 ++++++++++++++++++++++++++++++++++++++++++++-
 fs/xfs/xfs_icache.h       |   13 +
 fs/xfs/xfs_inode.c        |  102 ++++++++
 fs/xfs/xfs_inode.h        |   15 +
 fs/xfs/xfs_iomap.c        |   14 +
 fs/xfs/xfs_log_recover.c  |    7 +
 fs/xfs/xfs_mount.c        |   23 ++
 fs/xfs/xfs_mount.h        |   12 +
 fs/xfs/xfs_qm.c           |   29 ++
 fs/xfs/xfs_qm.h           |   17 +
 fs/xfs/xfs_qm_syscalls.c  |   20 ++
 fs/xfs/xfs_super.c        |   63 ++++-
 fs/xfs/xfs_trace.h        |   15 +
 17 files changed, 909 insertions(+), 32 deletions(-)

[PATCHSET 0/3] xfs: indirect health reporting

[Darrick J. Wong]

Hi all,

This series enables the XFS health reporting infrastructure to remember
indirect health concerns when resources are scarce.  For example, if a
scrub notices that there's something wrong with an inode's metadata but
memory reclaim needs to kill the incore inode, we ought to be able to
report that indirect observation in the AGI health report.

The first two patches in this series set that up, and the third one
provides a means for xfs_scrub to tell the kernel that it can forget the
indirect problem report.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=indirect-health-reporting

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=indirect-health-reporting
---
 fs/xfs/libxfs/xfs_fs.h        |    4 +-
 fs/xfs/libxfs/xfs_health.h    |   45 ++++++++++++++++++++++
 fs/xfs/libxfs/xfs_inode_buf.c |    2 -
 fs/xfs/scrub/health.c         |   83 +++++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/health.h         |    1 
 fs/xfs/scrub/repair.c         |    1 
 fs/xfs/scrub/scrub.c          |    6 +++
 fs/xfs/scrub/trace.h          |    3 +
 fs/xfs/xfs_health.c           |   25 ++++++++----
 fs/xfs/xfs_inode.c            |   30 +++++++++++++++
 fs/xfs/xfs_trace.h            |    1 
 11 files changed, 189 insertions(+), 12 deletions(-)

[PATCHSET v22 0/5] xfs: online repair hard problems

[Darrick J. Wong]

Hi all,

This series tackles hard problems for online repair to solve.  Hard
problems are defined as ones that cannot be done without running afoul
of the regular xfs locking and synchronization rules.

We solve these problems by adding ability to freeze the filesystem to
perform a scrub operation, and then we add a couple more repair
functions to deal with incorrect fs summary counters and rebuild reverse
mapping btrees.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-hard-problems

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-hard-problems

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-hard-problems
---
 fs/xfs/Makefile                  |    2 
 fs/xfs/libxfs/xfs_bmap.c         |   34 +
 fs/xfs/libxfs/xfs_bmap.h         |    8 
 fs/xfs/libxfs/xfs_fs.h           |    6 
 fs/xfs/scrub/bitmap.c            |   14 
 fs/xfs/scrub/bitmap.h            |    1 
 fs/xfs/scrub/common.c            |   91 +++
 fs/xfs/scrub/common.h            |    2 
 fs/xfs/scrub/fscounters.c        |   90 ++-
 fs/xfs/scrub/fscounters_repair.c |   63 ++
 fs/xfs/scrub/repair.c            |   27 +
 fs/xfs/scrub/repair.h            |   17 
 fs/xfs/scrub/rmap.c              |    6 
 fs/xfs/scrub/rmap_repair.c       | 1344 ++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/scrub.c             |   14 
 fs/xfs/scrub/scrub.h             |    1 
 fs/xfs/scrub/trace.h             |   51 +
 fs/xfs/xfs_icache.c              |    3 
 fs/xfs/xfs_icache.h              |    6 
 fs/xfs/xfs_mount.h               |    7 
 fs/xfs/xfs_super.c               |   47 +
 fs/xfs/xfs_trans.c               |    5 
 22 files changed, 1796 insertions(+), 43 deletions(-)
 create mode 100644 fs/xfs/scrub/fscounters_repair.c
 create mode 100644 fs/xfs/scrub/rmap_repair.c

[PATCHSET 0/4] xfs: widen BUI formats to support rt

[Darrick J. Wong]

Hi all,

Atomic extent swapping (and later, reverse mapping and reflink) on the
realtime device needs to be able to defer file mapping and extent
freeing work in much the same manner as is required on the data volume.
Make the BUI log items operate on rt extents in preparation for atomic
swapping and realtime rmap.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-bmap-intents

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-bmap-intents
---
 fs/xfs/libxfs/xfs_bmap.c       |   63 ++++++++++++++++++++++++----------------
 fs/xfs/libxfs/xfs_inode_fork.c |    9 ++++++
 fs/xfs/libxfs/xfs_inode_fork.h |    1 +
 fs/xfs/libxfs/xfs_log_format.h |    4 ++-
 fs/xfs/scrub/bmap.c            |    4 +--
 fs/xfs/scrub/rmap_repair.c     |   12 +-------
 fs/xfs/xfs_bmap_item.c         |   17 ++++++++++-
 fs/xfs/xfs_trace.h             |    8 +++--
 8 files changed, 74 insertions(+), 44 deletions(-)

[PATCHSET 0/2] xfs: support attrfork and unwritten BUIs

[Darrick J. Wong]

Hi all,

In preparation for atomic extent swapping and the online repair
functionality that wants atomic extent swaps, enhance the BUI code so
that we can support deferred work on the extended attribute fork and on
unwritten extents.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=expand-bmap-intent-usage

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=expand-bmap-intent-usage
---
 fs/xfs/libxfs/xfs_bmap.c |   44 ++++++++++++++++++--------------------------
 fs/xfs/libxfs/xfs_bmap.h |    4 ++--
 fs/xfs/xfs_bmap_item.c   |    2 +-
 fs/xfs/xfs_bmap_util.c   |    8 ++++----
 fs/xfs/xfs_reflink.c     |    6 +++---
 5 files changed, 28 insertions(+), 36 deletions(-)

[PATCHSET RFC v2 00/17] xfs: atomic file updates

[Darrick J. Wong]

Hi all,

This series creates a new log incompat feature and log intent items to
track high level progress of swapping ranges of two files and finish
interrupted work if the system goes down.  It then adds a new
FISWAPRANGE ioctl so that userspace can access the atomic extent
swapping feature.  With this feature, user programs will be able to
update files atomically by opening an O_TMPFILE, reflinking the source
file to it, making whatever updates they want to make, and then
atomically swap the changed bits back to the source file.  It even has
an optional ability to detect a changed source file and reject the
update.

The intent behind this new userspace functionality is to enable atomic
rewrites of arbitrary parts of individual files.  For years, application
programmers wanting to ensure the atomicity of a file update had to
write the changes to a new file in the same directory, fsync the new
file, rename the new file on top of the old filename, and then fsync the
directory.  People get it wrong all the time, and $fs hacks abound.

With atomic file updates, this is no longer necessary.  Programmers
create an O_TMPFILE, optionally FICLONE the file contents into the
temporary file, make whatever changes they want to the tempfile, and
FISWAPRANGE the contents from the tempfile into the regular file.  The
interface can optionally check the original file's [cm]time to reject
the swap operation if the file has been modified by.  There are no
fsyncs to take care of; no directory operations at all; and the fs will
take care of finishing the swap operation if the system goes down in the
middle of the swap.  Sample code can be found in the corresponding
changes to xfs_io to exercise the use case mentioned above.

Note that this function is /not/ the O_DIRECT atomic file writes concept
that has been floating around for years.  This is constructed entirely
in software, which means that there are no limitations other than the
regular filesystem limits.

As a side note, there's an extra motivation behind the kernel
functionality: online repair of file-based metadata.  The atomic file
swap is implemented as an atomic inode fork swap, which means that we
can implement online reconstruction of extended attributes and
directories by building a new one in another inode and atomically
swap the contents.

Next, we adapt the online filesystem repair code to use atomic extent
swapping.  This enables repair functions to construct a clean copy of a
directory, xattr information, realtime bitmaps, and realtime summary
information in a temporary inode.  If this completes successfully, the
new contents can be swapped atomically into the inode being repaired.
This is essential to avoid making corruption problems worse if the
system goes down in the middle of running repair.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=atomic-file-updates

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=atomic-file-updates

xfsdocs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=atomic-file-updates
---
 Documentation/filesystems/vfs.rst |   16 +
 fs/ioctl.c                        |   42 ++
 fs/remap_range.c                  |  283 +++++++++++++
 fs/xfs/Makefile                   |    3 
 fs/xfs/libxfs/xfs_defer.c         |   49 ++
 fs/xfs/libxfs/xfs_defer.h         |   11 -
 fs/xfs/libxfs/xfs_errortag.h      |    4 
 fs/xfs/libxfs/xfs_format.h        |   38 ++
 fs/xfs/libxfs/xfs_fs.h            |    2 
 fs/xfs/libxfs/xfs_log_format.h    |   63 +++
 fs/xfs/libxfs/xfs_log_recover.h   |    4 
 fs/xfs/libxfs/xfs_sb.c            |    2 
 fs/xfs/libxfs/xfs_swapext.c       |  793 +++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_swapext.h       |   89 ++++
 fs/xfs/xfs_bmap_item.c            |   13 -
 fs/xfs/xfs_bmap_util.c            |  595 ----------------------------
 fs/xfs/xfs_bmap_util.h            |    3 
 fs/xfs/xfs_error.c                |    3 
 fs/xfs/xfs_extfree_item.c         |    2 
 fs/xfs/xfs_file.c                 |   49 ++
 fs/xfs/xfs_inode.c                |   13 +
 fs/xfs/xfs_inode.h                |    1 
 fs/xfs/xfs_ioctl.c                |  102 +----
 fs/xfs/xfs_ioctl.h                |    4 
 fs/xfs/xfs_ioctl32.c              |    8 
 fs/xfs/xfs_log.c                  |   10 
 fs/xfs/xfs_log_recover.c          |   41 ++
 fs/xfs/xfs_mount.c                |  119 ++++++
 fs/xfs/xfs_mount.h                |    2 
 fs/xfs/xfs_refcount_item.c        |    2 
 fs/xfs/xfs_rmap_item.c            |    2 
 fs/xfs/xfs_super.c                |   26 +
 fs/xfs/xfs_swapext_item.c         |  649 ++++++++++++++++++++++++++++++
 fs/xfs/xfs_swapext_item.h         |   61 +++
 fs/xfs/xfs_trace.c                |    1 
 fs/xfs/xfs_trace.h                |  116 +++++
 fs/xfs/xfs_xchgrange.c            |  721 ++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_xchgrange.h            |   30 +
 include/linux/fs.h                |   14 +
 include/uapi/linux/fiexchange.h   |  101 +++++
 40 files changed, 3363 insertions(+), 724 deletions(-)
 create mode 100644 fs/xfs/libxfs/xfs_swapext.c
 create mode 100644 fs/xfs/libxfs/xfs_swapext.h
 create mode 100644 fs/xfs/xfs_swapext_item.c
 create mode 100644 fs/xfs/xfs_swapext_item.h
 create mode 100644 fs/xfs/xfs_xchgrange.c
 create mode 100644 fs/xfs/xfs_xchgrange.h
 create mode 100644 include/uapi/linux/fiexchange.h

[PATCHSET v22 0/3] xfs: online repair of realtime summaries

[Darrick J. Wong]

Hi all,

This series uses the atomic extent swapping code to enable safe
reconstruction of the realtime summary file.

First, we enable online repair to allocate a temporary file on the
filesystem that can be used by online repair to stage reconstructed
metadata.  Next, we walk the realtime bitmap to compute a new summary in
this temporary file.  Finally, we swap the contents of the two files
with the atomic extent swap feature.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rtsummary

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-rtsummary
---
 fs/xfs/Makefile                 |    4 
 fs/xfs/scrub/repair.c           |  315 +++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/repair.h           |   19 ++
 fs/xfs/scrub/rtsummary.c        |   22 +++
 fs/xfs/scrub/rtsummary_repair.c |   65 ++++++++
 fs/xfs/scrub/scrub.c            |    8 +
 fs/xfs/scrub/scrub.h            |    4 
 fs/xfs/xfs_export.c             |    2 
 fs/xfs/xfs_inode.c              |    3 
 fs/xfs/xfs_inode.h              |    1 
 fs/xfs/xfs_itable.c             |    8 +
 fs/xfs/xfs_xchgrange.c          |    2 
 fs/xfs/xfs_xchgrange.h          |    2 
 13 files changed, 447 insertions(+), 8 deletions(-)
 create mode 100644 fs/xfs/scrub/rtsummary_repair.c

[PATCHSET v22 0/3] xfs: online repair of extended attributes

[Darrick J. Wong]

Hi all,

This series employs atomic extent swapping to enable safe reconstruction
of extended attribute data.  Extended attribute repair consists of four
main parts:

First, we walk the existing attributes to salvage as many of them as we
can, by adding them as new attributes attached to the repair tempfile.

Second, we prepare the temp file by changing the inode owner field in
the attr block headers.

Third, we use atomic extent swaps to exchange the entire attribute fork
between the two files.

Finally, we tear down the old attr data (which is now in the temporary
file) as carefully as we can.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-xattrs
---
 fs/xfs/Makefile            |    2 
 fs/xfs/libxfs/xfs_attr.c   |    2 
 fs/xfs/libxfs/xfs_attr.h   |    1 
 fs/xfs/scrub/array.c       |   24 +
 fs/xfs/scrub/array.h       |    2 
 fs/xfs/scrub/attr.c        |    9 
 fs/xfs/scrub/attr.h        |   10 
 fs/xfs/scrub/attr_repair.c | 1393 ++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/bitmap.c      |   22 +
 fs/xfs/scrub/bitmap.h      |    1 
 fs/xfs/scrub/blob.c        |  167 +++++
 fs/xfs/scrub/blob.h        |   26 +
 fs/xfs/scrub/dabtree.c     |   16 +
 fs/xfs/scrub/dabtree.h     |    1 
 fs/xfs/scrub/repair.c      |  136 ++++
 fs/xfs/scrub/repair.h      |    9 
 fs/xfs/scrub/scrub.c       |    2 
 fs/xfs/scrub/scrub.h       |    3 
 fs/xfs/scrub/trace.c       |    1 
 fs/xfs/scrub/trace.h       |   86 +++
 fs/xfs/xfs_buf.c           |    9 
 fs/xfs/xfs_buf.h           |    9 
 22 files changed, 1927 insertions(+), 4 deletions(-)
 create mode 100644 fs/xfs/scrub/attr_repair.c
 create mode 100644 fs/xfs/scrub/blob.c
 create mode 100644 fs/xfs/scrub/blob.h

[PATCHSET v22 0/4] xfs: online repair of directories

[Darrick J. Wong]

Hi all,

This series employs atomic extent swapping to enable safe reconstruction
of directory data.  Directory repair consists of five main parts:

First, we walk the existing directory to salvage as many entries as we
can, by adding them as new directory entries to the repair temp dir.

Second, we validate the parent pointer found in the directory.  If one
was not found, we scan the entire filesystem looking for a potential
parent.

Third, we prepare the temp file by changing the inode owner field in
the directory block headers.

Fourth, we use atomic extent swaps to exchange the entire data fork
between the two directories.

Finally, we add the ability to inactivate directories by directly
freeing all the data fork blocks.  This does not change anything with
normal directories, since they must still unlink and shrink one entry at
a time.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs
---
 fs/xfs/Makefile              |    2 
 fs/xfs/scrub/common.c        |    4 
 fs/xfs/scrub/dir.c           |   49 ++
 fs/xfs/scrub/dir_repair.c    | 1176 ++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/parent.c        |   39 +
 fs/xfs/scrub/parent.h        |   11 
 fs/xfs/scrub/parent_repair.c |  439 ++++++++++++++++
 fs/xfs/scrub/repair.c        |  166 ++++++
 fs/xfs/scrub/repair.h        |    6 
 fs/xfs/scrub/scrub.c         |   10 
 fs/xfs/scrub/scrub.h         |    4 
 fs/xfs/scrub/trace.h         |  123 ++++
 fs/xfs/xfs_inode.c           |   55 ++
 fs/xfs/xfs_inode.h           |    1 
 14 files changed, 2077 insertions(+), 8 deletions(-)
 create mode 100644 fs/xfs/scrub/dir_repair.c
 create mode 100644 fs/xfs/scrub/parent.h
 create mode 100644 fs/xfs/scrub/parent_repair.c

[PATCHSET v2 00/20] xfs: hoist inode operations to libxfs

[Darrick J. Wong]

Hi all,

This series hoists inode creation, renaming, and deletion operations to
libxfs in anticipation of the metadata inode directory feature, which
maintains a directory tree of metadata inodes.  This will be necessary
for further enhancements to the realtime feature, subvolume support.

There aren't supposed to be any functional changes in this intense
refactoring -- we just split the functions into pieces that are generic
and pieces that are specific to libxfs clients.  As a bonus, we can
remove various open-coded pieces of mkfs.xfs and xfs_repair when this
series gets to xfsprogs.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=inode-refactor

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=inode-refactor
---
 fs/xfs/Makefile                 |    1 
 fs/xfs/libxfs/xfs_bmap.c        |   44 +
 fs/xfs/libxfs/xfs_bmap.h        |    3 
 fs/xfs/libxfs/xfs_dir2.c        |  474 +++++++++++++++
 fs/xfs/libxfs/xfs_dir2.h        |   19 +
 fs/xfs/libxfs/xfs_inode_util.c  |  859 +++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_inode_util.h  |   74 ++
 fs/xfs/libxfs/xfs_shared.h      |    8 
 fs/xfs/libxfs/xfs_trans_inode.c |    2 
 fs/xfs/scrub/repair.c           |   13 
 fs/xfs/xfs_inode.c              | 1255 ++-------------------------------------
 fs/xfs/xfs_inode.h              |   31 -
 fs/xfs/xfs_ioctl.c              |   59 --
 fs/xfs/xfs_iops.c               |   45 +
 fs/xfs/xfs_linux.h              |    2 
 fs/xfs/xfs_qm.c                 |    6 
 fs/xfs/xfs_reflink.h            |    6 
 fs/xfs/xfs_symlink.c            |   16 
 fs/xfs/xfs_trans.h              |    1 
 19 files changed, 1594 insertions(+), 1324 deletions(-)
 create mode 100644 fs/xfs/libxfs/xfs_inode_util.c
 create mode 100644 fs/xfs/libxfs/xfs_inode_util.h

[PATCHSET v2 00/15] xfs: metadata inode directories

[Darrick J. Wong]

Hi all,

This series delivers a new feature -- metadata inode directories.  This
is a separate directory tree (rooted in the superblock) that contains
only inodes that contain filesystem metadata.  Different metadata
objects can be looked up with regular paths.

We start by creating xfs_imeta_* functions to mediate access to metadata
inode pointers.  This enables the imeta code to abstract inode pointers,
whether they're the classic five in the superblock, or the much more
complex directory tree.  All current users of metadata inodes (rt+quota)
are converted to use the boilerplate code.

Next, we define the metadir on-disk format, which consists of marking
inodes with a new iflag that says they're metadata.  This we use to
prevent bulkstat and friends from ever getting their hands on fs
metadata.

Finally, we implement metadir operations so that clients can create,
delete, zap, and look up metadata inodes by path.  Beware that much of
this code is only lightly used, because the five current users of
metadata inodes don't tend to change them very often.  This is likely to
change if and when the subvolume and multiple-rt-volume features get
written/merged/etc.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=metadir

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=metadir

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=metadir

xfsdocs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=metadir
---
 fs/xfs/Makefile                |    1 
 fs/xfs/libxfs/xfs_dir2.c       |    1 
 fs/xfs/libxfs/xfs_format.h     |   55 ++
 fs/xfs/libxfs/xfs_ialloc.c     |   47 +-
 fs/xfs/libxfs/xfs_ialloc.h     |    2 
 fs/xfs/libxfs/xfs_imeta.c      |  959 ++++++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_imeta.h      |   73 +++
 fs/xfs/libxfs/xfs_inode_buf.c  |   69 +++
 fs/xfs/libxfs/xfs_inode_buf.h  |    3 
 fs/xfs/libxfs/xfs_inode_util.c |    4 
 fs/xfs/libxfs/xfs_sb.c         |   31 +
 fs/xfs/libxfs/xfs_trans_resv.c |   83 +++
 fs/xfs/libxfs/xfs_trans_resv.h |    2 
 fs/xfs/libxfs/xfs_types.c      |    7 
 fs/xfs/scrub/common.c          |    3 
 fs/xfs/scrub/inode_repair.c    |    6 
 fs/xfs/scrub/scrub.c           |    1 
 fs/xfs/xfs_icache.c            |   38 ++
 fs/xfs/xfs_inode.c             |  132 ++++++
 fs/xfs/xfs_inode.h             |    5 
 fs/xfs/xfs_iops.c              |   34 +
 fs/xfs/xfs_mount.c             |   39 ++
 fs/xfs/xfs_mount.h             |    1 
 fs/xfs/xfs_qm.c                |  192 +++++---
 fs/xfs/xfs_qm_syscalls.c       |    4 
 fs/xfs/xfs_rtalloc.c           |   14 -
 fs/xfs/xfs_super.c             |    4 
 fs/xfs/xfs_trace.h             |   81 +++
 28 files changed, 1786 insertions(+), 105 deletions(-)
 create mode 100644 fs/xfs/libxfs/xfs_imeta.c
 create mode 100644 fs/xfs/libxfs/xfs_imeta.h

[PATCHSET 00/14] xfs: refactor btrees to support records in inode root

[Darrick J. Wong]

Hi all,

This series prepares the btree code to support realtime reverse mapping
btrees by refactoring xfs_ifork_realloc to be fed a per-btree ops
structure so that it can handle multiple types of inode-rooted btrees.
It moves on to refactoring the btree code to use the new realloc
routines and to support storing btree rcords in the inode root, because
the current bmbt code does not support this.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=btree-ifork-records

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=btree-ifork-records
---
 fs/xfs/libxfs/xfs_alloc_btree.c    |    6 -
 fs/xfs/libxfs/xfs_alloc_btree.h    |    3 
 fs/xfs/libxfs/xfs_attr_leaf.c      |    8 -
 fs/xfs/libxfs/xfs_bmap.c           |   60 ++----
 fs/xfs/libxfs/xfs_bmap_btree.c     |   94 ++++++++-
 fs/xfs/libxfs/xfs_bmap_btree.h     |  219 +++++++++++++++------
 fs/xfs/libxfs/xfs_btree.c          |  382 ++++++++++++++++++++++++++++--------
 fs/xfs/libxfs/xfs_btree.h          |    4 
 fs/xfs/libxfs/xfs_btree_staging.c  |    4 
 fs/xfs/libxfs/xfs_ialloc.c         |    4 
 fs/xfs/libxfs/xfs_ialloc_btree.c   |    6 -
 fs/xfs/libxfs/xfs_ialloc_btree.h   |    3 
 fs/xfs/libxfs/xfs_inode_fork.c     |  168 +++++++---------
 fs/xfs/libxfs/xfs_inode_fork.h     |   27 ++-
 fs/xfs/libxfs/xfs_refcount_btree.c |    5 
 fs/xfs/libxfs/xfs_refcount_btree.h |    3 
 fs/xfs/libxfs/xfs_rmap_btree.c     |    7 -
 fs/xfs/libxfs/xfs_rmap_btree.h     |    3 
 fs/xfs/libxfs/xfs_sb.c             |   16 +-
 fs/xfs/libxfs/xfs_trans_resv.c     |    2 
 fs/xfs/scrub/bmap_repair.c         |    4 
 fs/xfs/scrub/inode_repair.c        |    2 
 fs/xfs/xfs_xchgrange.c             |    4 
 23 files changed, 705 insertions(+), 329 deletions(-)

[PATCHSET 00/13] xfs: support dynamic btree cursor height

[Darrick J. Wong]

Hi all,

Rearrange the incore btree cursor so that we can support btrees of any
height.  This will become necessary for realtime rmap and reflink since
we'd like to handle tall trees without bloating the ag btree cursors.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=btree-dynamic-depth

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=btree-dynamic-depth
---
 fs/xfs/libxfs/xfs_ag_resv.c        |    4 -
 fs/xfs/libxfs/xfs_alloc.c          |   18 +--
 fs/xfs/libxfs/xfs_alloc_btree.c    |    8 -
 fs/xfs/libxfs/xfs_bmap.c           |   24 ++--
 fs/xfs/libxfs/xfs_bmap_btree.c     |    7 -
 fs/xfs/libxfs/xfs_btree.c          |  236 ++++++++++++++++++++++--------------
 fs/xfs/libxfs/xfs_btree.h          |   50 +++++---
 fs/xfs/libxfs/xfs_btree_staging.c  |    6 -
 fs/xfs/libxfs/xfs_ialloc_btree.c   |    7 -
 fs/xfs/libxfs/xfs_refcount_btree.c |    6 -
 fs/xfs/libxfs/xfs_rmap_btree.c     |   60 ++++++---
 fs/xfs/libxfs/xfs_trans_resv.c     |   18 ++-
 fs/xfs/libxfs/xfs_trans_space.h    |    7 +
 fs/xfs/scrub/agheader.c            |   13 +-
 fs/xfs/scrub/agheader_repair.c     |    8 +
 fs/xfs/scrub/bitmap.c              |   16 +-
 fs/xfs/scrub/bmap.c                |    2 
 fs/xfs/scrub/btree.c               |  124 ++++++++++---------
 fs/xfs/scrub/btree.h               |   17 ++-
 fs/xfs/scrub/repair.h              |    3 
 fs/xfs/scrub/trace.c               |    4 -
 fs/xfs/scrub/trace.h               |   10 +-
 fs/xfs/xfs_super.c                 |   11 --
 fs/xfs/xfs_trace.h                 |    2 
 24 files changed, 383 insertions(+), 278 deletions(-)

[PATCHSET 0/2] xfs: refactor realtime meta inode locking

[Darrick J. Wong]

Hi all,

Replace all the open-coded locking of realtime metadata inodes with a
single rtlock function that can lock all the pieces that the caller
wants in a single call.  This will be important for maintaining correct
locking order later when we start adding more realtime metadata inodes.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=refactor-rt-locking

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=refactor-rt-locking
---
 fs/xfs/libxfs/xfs_bmap.c |    7 ++-----
 fs/xfs/scrub/bmap.c      |    4 ++++
 fs/xfs/scrub/common.c    |   32 ++++++++++++++++++++++++++++++
 fs/xfs/scrub/common.h    |    2 ++
 fs/xfs/scrub/rtbitmap.c  |    9 ++------
 fs/xfs/scrub/rtsummary.c |   26 +++++-------------------
 fs/xfs/scrub/scrub.c     |    3 ++-
 fs/xfs/scrub/scrub.h     |    8 ++++++++
 fs/xfs/xfs_bmap_util.c   |    5 +----
 fs/xfs/xfs_fsmap.c       |    4 ++--
 fs/xfs/xfs_rtalloc.c     |   49 ++++++++++++++++++++++++++++++++++++++--------
 fs/xfs/xfs_rtalloc.h     |    9 ++++++++
 12 files changed, 110 insertions(+), 48 deletions(-)

[PATCHSET 0/2] xfs: enable in-core block reservation for rt metadata

[Darrick J. Wong]

Hi all,

In preparation for adding reverse mapping and refcounting to the
realtime device, enhance the per-AG block reservation system for use
with realtime metadata.  This effectively allows us to pre-allocate
space for the rmap and refcount btrees in the same manner as we do for
the data device counterparts, which is how we avoid ENOSPC failures when
space is low but we've already committed to a COW operation.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=reserve-rt-metadata-space

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=reserve-rt-metadata-space
---
 fs/xfs/libxfs/xfs_ag_resv.c |  116 +++++++++++++++++++++++++++++++++++++------
 fs/xfs/libxfs/xfs_ag_resv.h |    6 +-
 fs/xfs/libxfs/xfs_types.h   |    1 
 fs/xfs/scrub/fscounters.c   |    3 +
 fs/xfs/xfs_fsops.c          |    9 +++
 fs/xfs/xfs_mount.h          |   25 ++++++---
 fs/xfs/xfs_rtalloc.c        |   13 +++++
 fs/xfs/xfs_trace.h          |   43 ++++++++++++++--
 8 files changed, 183 insertions(+), 33 deletions(-)

[PATCHSET 0/3] xfs: widen EFI format to support rt

[Darrick J. Wong]

Hi all,

Realtime reverse mapping (and beyond that, realtime reflink) needs to be
able to defer file mapping and extent freeing work in much the same
manner as is required on the data volume.  Make the extent freeing log
items operate on rt extents in preparation for realtime rmap.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-extfree-intents

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-extfree-intents
---
 fs/xfs/libxfs/xfs_alloc.c      |    1 +
 fs/xfs/libxfs/xfs_bmap.c       |   37 ++++++++++++++++------
 fs/xfs/libxfs/xfs_bmap.h       |    8 +++--
 fs/xfs/libxfs/xfs_bmap_btree.c |    2 +
 fs/xfs/libxfs/xfs_ialloc.c     |    4 +-
 fs/xfs/libxfs/xfs_log_format.h |    3 ++
 fs/xfs/libxfs/xfs_refcount.c   |    8 +++--
 fs/xfs/libxfs/xfs_rtbitmap.c   |    4 ++
 fs/xfs/scrub/alloc_repair.c    |    3 +-
 fs/xfs/scrub/repair.c          |    5 ++-
 fs/xfs/xfs_extfree_item.c      |   67 +++++++++++++++++++++++++++++++++++-----
 fs/xfs/xfs_reflink.c           |    2 +
 12 files changed, 112 insertions(+), 32 deletions(-)

[PATCHSET v11 00/27] xfs: realtime reverse-mapping support

[Darrick J. Wong]

Hi all,

This is the latest revision of a patchset that adds to XFS kernel
support for reverse mapping for the realtime device.  This time around
I've fixed some of the bitrot that I've noticed over the past few
months, and most notably have converted rtrmapbt to use the metadata
inode directory feature instead of burning more space in the superblock.

At the beginning of the set are patches to implement storing B+tree
leaves in an inode root, since the realtime rmapbt is rooted in an
inode, unlike the regular rmapbt which is rooted in an AG block.
Prior to this, the only btree that could be rooted in the inode fork
was the block mapping btree; if all the extent records fit in the
inode, format would be switched from 'btree' to 'extents'.

The next few patches widen the reverse mapping routines to fit the
64-bit numbers required to store information about the realtime
device and establish a new b+tree type (rtrmapbt) for the realtime
variant of the rmapbt.  After that are a few patches to handle rooting
the rtrmapbt in a specific inode that's referenced by the superblock.

Finally, there are patches to implement GETFSMAP with the rtrmapbt and
scrub functionality for the rtrmapbt and rtbitmap; and then wire up the
online scrub functionality.  We also enhance EFIs to support tracking
freeing of realtime extents so that when rmap is turned on we can
maintain the same order of operations as the regular rmap code.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-rmap

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-rmap

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-rmap

xfsdocs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=realtime-rmap
---
 fs/xfs/Makefile                  |    4 
 fs/xfs/libxfs/xfs_ag_resv.c      |    5 
 fs/xfs/libxfs/xfs_bmap.c         |   29 +
 fs/xfs/libxfs/xfs_btree.c        |  108 +++++
 fs/xfs/libxfs/xfs_btree.h        |    9 
 fs/xfs/libxfs/xfs_format.h       |   76 ++++
 fs/xfs/libxfs/xfs_fs.h           |    4 
 fs/xfs/libxfs/xfs_health.h       |    4 
 fs/xfs/libxfs/xfs_imeta.c        |    2 
 fs/xfs/libxfs/xfs_imeta.h        |    1 
 fs/xfs/libxfs/xfs_inode_buf.c    |    6 
 fs/xfs/libxfs/xfs_inode_fork.c   |   13 +
 fs/xfs/libxfs/xfs_log_format.h   |    4 
 fs/xfs/libxfs/xfs_refcount.c     |    6 
 fs/xfs/libxfs/xfs_rmap.c         |  377 +++++++++++-------
 fs/xfs/libxfs/xfs_rmap.h         |   39 +-
 fs/xfs/libxfs/xfs_rtrmap_btree.c |  778 ++++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_rtrmap_btree.h |  195 ++++++++++
 fs/xfs/libxfs/xfs_sb.c           |    6 
 fs/xfs/libxfs/xfs_shared.h       |    1 
 fs/xfs/libxfs/xfs_swapext.c      |    3 
 fs/xfs/libxfs/xfs_trans_resv.c   |   17 +
 fs/xfs/libxfs/xfs_trans_space.h  |   13 +
 fs/xfs/libxfs/xfs_types.h        |    6 
 fs/xfs/scrub/alloc_repair.c      |    6 
 fs/xfs/scrub/bmap.c              |   76 +++-
 fs/xfs/scrub/bmap_repair.c       |   87 ++++
 fs/xfs/scrub/common.c            |   37 ++
 fs/xfs/scrub/common.h            |    8 
 fs/xfs/scrub/health.c            |    1 
 fs/xfs/scrub/repair.c            |   95 +++++
 fs/xfs/scrub/repair.h            |   13 +
 fs/xfs/scrub/rmap_repair.c       |   21 +
 fs/xfs/scrub/rtbitmap.c          |   59 +++
 fs/xfs/scrub/rtbitmap_repair.c   |  252 ++++++++++++
 fs/xfs/scrub/rtrmap.c            |  147 +++++++
 fs/xfs/scrub/rtrmap_repair.c     |  689 ++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/scrub.c             |   11 -
 fs/xfs/scrub/scrub.h             |   11 +
 fs/xfs/scrub/trace.c             |    2 
 fs/xfs/scrub/trace.h             |   87 ++++
 fs/xfs/xfs_buf_item_recover.c    |    4 
 fs/xfs/xfs_fsmap.c               |   66 +++
 fs/xfs/xfs_health.c              |    4 
 fs/xfs/xfs_inode.c               |    9 
 fs/xfs/xfs_inode_item.c          |    2 
 fs/xfs/xfs_inode_item_recover.c  |   33 +-
 fs/xfs/xfs_mount.c               |    2 
 fs/xfs/xfs_mount.h               |    4 
 fs/xfs/xfs_ondisk.h              |    3 
 fs/xfs/xfs_rmap_item.c           |   31 +-
 fs/xfs/xfs_rtalloc.c             |   55 ++-
 fs/xfs/xfs_rtalloc.h             |    4 
 fs/xfs/xfs_super.c               |    6 
 fs/xfs/xfs_trace.h               |   39 +-
 55 files changed, 3286 insertions(+), 284 deletions(-)
 create mode 100644 fs/xfs/libxfs/xfs_rtrmap_btree.c
 create mode 100644 fs/xfs/libxfs/xfs_rtrmap_btree.h
 create mode 100644 fs/xfs/scrub/rtbitmap_repair.c
 create mode 100644 fs/xfs/scrub/rtrmap.c
 create mode 100644 fs/xfs/scrub/rtrmap_repair.c

[PATCHSET RFC 0/4] xfs: noalloc allocation groups

[Darrick J. Wong]

Hi all,

This series creates a new NOALLOC flag for allocation groups that causes
the block and inode allocators to look elsewhere when trying to
allocate resources.  This is either the first part of a patchset to
implement online shrinking (set noalloc on the last AGs, run fsr to move
the files and directories) or freeze-free rmapbt rebuilding (set
noalloc to prevent creation of new mappings, then hook deletion of old
mappings).  This is still totally a research project.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=noalloc-ags

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=noalloc-ags
---
 fs/xfs/libxfs/xfs_ag.c      |  120 +++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_ag.h      |    4 +
 fs/xfs/libxfs/xfs_ag_resv.c |   26 ++++++++-
 fs/xfs/libxfs/xfs_fs.h      |    5 ++
 fs/xfs/libxfs/xfs_ialloc.c  |    5 +-
 fs/xfs/scrub/fscounters.c   |    3 +
 fs/xfs/xfs_fsops.c          |   14 +++++
 fs/xfs/xfs_ioctl.c          |    4 +
 fs/xfs/xfs_mount.h          |   11 ++--
 fs/xfs/xfs_super.c          |    1 
 fs/xfs/xfs_trace.h          |   46 ++++++++++++++++
 11 files changed, 227 insertions(+), 12 deletions(-)

[PATCHSET 00/31] xfs: reflink on the realtime device

[Darrick J. Wong]

Hi all,

This patchset enables use of the file data block sharing feature (i.e.
reflink) on the realtime device.  It follows the same basic sequence as
the realtime rmap series -- first a few cleanups; then widening of the
API parameters; and introduction of the new btree format and inode fork
format.  Next comes enabling CoW and remapping for the rt device; new
scrub, repair, and health reporting code; and at the end we implement
some code to lengthen write requests so that rt extents are always CoWed
fully.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-reflink

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-reflink

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-reflink

xfsdocs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=realtime-reflink
---
 fs/xfs/Makefile                      |    3 
 fs/xfs/libxfs/xfs_ag_resv.c          |    5 
 fs/xfs/libxfs/xfs_bmap.c             |   10 
 fs/xfs/libxfs/xfs_btree.c            |    8 
 fs/xfs/libxfs/xfs_btree.h            |   14 -
 fs/xfs/libxfs/xfs_format.h           |   53 ++-
 fs/xfs/libxfs/xfs_fs.h               |    4 
 fs/xfs/libxfs/xfs_health.h           |    4 
 fs/xfs/libxfs/xfs_imeta.c            |    2 
 fs/xfs/libxfs/xfs_imeta.h            |    1 
 fs/xfs/libxfs/xfs_inode_buf.c        |   20 +
 fs/xfs/libxfs/xfs_inode_fork.c       |   13 +
 fs/xfs/libxfs/xfs_log_format.h       |    5 
 fs/xfs/libxfs/xfs_refcount.c         |  693 ++++++++++++++++++++-------------
 fs/xfs/libxfs/xfs_refcount.h         |   34 +-
 fs/xfs/libxfs/xfs_rmap.c             |    4 
 fs/xfs/libxfs/xfs_rtrefcount_btree.c |  704 ++++++++++++++++++++++++++++++++++
 fs/xfs/libxfs/xfs_rtrefcount_btree.h |  184 +++++++++
 fs/xfs/libxfs/xfs_rtrmap_btree.c     |   44 ++
 fs/xfs/libxfs/xfs_sb.c               |    8 
 fs/xfs/libxfs/xfs_shared.h           |    1 
 fs/xfs/libxfs/xfs_trans_resv.c       |   25 +
 fs/xfs/libxfs/xfs_types.h            |    5 
 fs/xfs/scrub/bmap.c                  |   12 +
 fs/xfs/scrub/bmap_repair.c           |    3 
 fs/xfs/scrub/common.c                |    8 
 fs/xfs/scrub/common.h                |    6 
 fs/xfs/scrub/health.c                |    1 
 fs/xfs/scrub/inode.c                 |    6 
 fs/xfs/scrub/inode_repair.c          |    6 
 fs/xfs/scrub/repair.c                |   20 +
 fs/xfs/scrub/repair.h                |    5 
 fs/xfs/scrub/rmap.c                  |    4 
 fs/xfs/scrub/rmap_repair.c           |   23 +
 fs/xfs/scrub/rtbitmap.c              |    1 
 fs/xfs/scrub/rtbitmap_repair.c       |   13 +
 fs/xfs/scrub/rtrefcount.c            |  574 ++++++++++++++++++++++++++++
 fs/xfs/scrub/rtrefcount_repair.c     |  706 ++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/rtrmap.c                |   52 ++-
 fs/xfs/scrub/rtrmap_repair.c         |    3 
 fs/xfs/scrub/scrub.c                 |    7 
 fs/xfs/scrub/scrub.h                 |   13 +
 fs/xfs/scrub/trace.h                 |   27 +
 fs/xfs/xfs_bmap_util.c               |   60 ++-
 fs/xfs/xfs_buf_item_recover.c        |    4 
 fs/xfs/xfs_fsmap.c                   |   16 -
 fs/xfs/xfs_health.c                  |    4 
 fs/xfs/xfs_inode.c                   |    7 
 fs/xfs/xfs_inode_item.c              |    2 
 fs/xfs/xfs_inode_item_recover.c      |    5 
 fs/xfs/xfs_ioctl.c                   |   15 -
 fs/xfs/xfs_mount.c                   |    2 
 fs/xfs/xfs_mount.h                   |    4 
 fs/xfs/xfs_ondisk.h                  |    3 
 fs/xfs/xfs_refcount_item.c           |   48 ++
 fs/xfs/xfs_reflink.c                 |  163 +++++---
 fs/xfs/xfs_reflink.h                 |    3 
 fs/xfs/xfs_rtalloc.c                 |   33 ++
 fs/xfs/xfs_rtalloc.h                 |    4 
 fs/xfs/xfs_super.c                   |    5 
 fs/xfs/xfs_trace.h                   |  107 +++--
 61 files changed, 3308 insertions(+), 511 deletions(-)
 create mode 100644 fs/xfs/libxfs/xfs_rtrefcount_btree.c
 create mode 100644 fs/xfs/libxfs/xfs_rtrefcount_btree.h
 create mode 100644 fs/xfs/scrub/rtrefcount.c
 create mode 100644 fs/xfs/scrub/rtrefcount_repair.c

[PATCHSET 0/7] xfs: reflink with large realtime extents

[Darrick J. Wong]

Hi all,

Now that we've landed support for reflink on the realtime device for
cases where the rt extent size is the same as the fs block size, enhance
the reflink code further to support cases where the rt extent size is a
power-of-two multiple of the fs block size.  This enables us to do data
block sharing (for example) for much larger allocation units by dirtying
pagecache around shared extents and expanding writeback to write back
shared extents fully.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-reflink-extsize
---
 fs/remap_range.c     |   37 +++++++---
 fs/xfs/xfs_aops.c    |   43 +++++++++---
 fs/xfs/xfs_file.c    |  174 +++++++++++++++++++++++++++++++++++++++++++-----
 fs/xfs/xfs_inode.h   |    2 +
 fs/xfs/xfs_iops.c    |   15 ++++
 fs/xfs/xfs_reflink.c |  181 +++++++++++++++++++++++++++++++++++++++++++++++++-
 fs/xfs/xfs_reflink.h |    9 ++
 fs/xfs/xfs_super.c   |   11 ++-
 include/linux/fs.h   |    5 +
 9 files changed, 429 insertions(+), 48 deletions(-)

[PATCHSET v3 0/2] xfs_db: add minimal directory navigation

[Darrick J. Wong]

Hi all,

Improve the usability of xfs_db by enabling users to navigate to inodes
by path and to list the contents of directories.

v2: Various cleanups and reorganizing suggested by dchinner
v3: Rebase to 5.10-rc0

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=xfs_db-directory-navigation

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=xfs_db-directory-navigation
---
 db/Makefile              |    3 
 db/command.c             |    1 
 db/command.h             |    1 
 db/namei.c               |  612 ++++++++++++++++++++++++++++++++++++++++++++++
 libxfs/libxfs_api_defs.h |    1 
 man/man8/xfs_db.8        |   20 ++
 6 files changed, 637 insertions(+), 1 deletion(-)
 create mode 100644 db/namei.c

[PATCHSET 0/3] xfs: add the ability to flag a fs for repair

[Darrick J. Wong]

Hi all,

Add a new incompat feature flag so that we can force a sysadmin to run
xfs_repair on a filesystem before mounting.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=needsrepair

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=needsrepair

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=needsrepair
---
 db/check.c           |    5 ++
 db/sb.c              |  158 ++++++++++++++++++++++++++++++++++++++++++++++++--
 db/xfs_admin.sh      |    6 ++
 man/man8/xfs_admin.8 |   15 +++++
 man/man8/xfs_db.8    |    5 ++
 repair/agheader.c    |   11 +++
 scrub/inodes.c       |   16 +++++
 7 files changed, 208 insertions(+), 8 deletions(-)

[PATCHSET 0/2] xfs_admin: support upgrading v5 filesystems

[Darrick J. Wong]

Hi all,

Extend xfs_db and xfs_admin to support adding the inode btree counter and
bigtime features to an existing v5 filesystem.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=fs-upgrades
---
 db/sb.c              |   38 ++++++++++++++++++++++++++++++++++++++
 man/man8/xfs_admin.8 |   12 ++++++++++++
 man/man8/xfs_db.8    |    7 +++++++
 3 files changed, 57 insertions(+)

[PATCHSET v2 0/2] xfs_repair: rebuild inode fork mappings

[Darrick J. Wong]

Hi all,

Add the ability to regenerate inode fork mappings if the rmapbt
otherwise looks ok.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-rebuild-forks
---
 include/xfs_trans.h      |    3 
 libxfs/libxfs_api_defs.h |   12 +
 libxfs/trans.c           |   48 ++++
 repair/Makefile          |    2 
 repair/agbtree.c         |    2 
 repair/bmap_repair.c     |  602 ++++++++++++++++++++++++++++++++++++++++++++++
 repair/bmap_repair.h     |   13 +
 repair/bulkload.c        |  167 +++++++++++++
 repair/bulkload.h        |   17 +
 repair/dino_chunks.c     |    5 
 repair/dinode.c          |  206 ++++++++++------
 repair/dinode.h          |    7 -
 repair/rmap.c            |    2 
 repair/rmap.h            |    1 
 14 files changed, 1006 insertions(+), 81 deletions(-)
 create mode 100644 repair/bmap_repair.c
 create mode 100644 repair/bmap_repair.h

[PATCHSET 0/2] xfsprogs: packaging cleanups

[Darrick J. Wong]

Hi all,

Introduce flatpak build system support, forward-port debian packages to
compat 11 to take advantage of systemd helper scripts.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=packaging-cleanups
---
 debian/compat                       |    2 +-
 debian/rules                        |    1 +
 scrub/Makefile                      |    3 ++-
 scrub/xfs_scrub_all_fail            |   24 ++++++++++++++++++++++++
 scrub/xfs_scrub_all_fail.service.in |   11 +++++++++++
 5 files changed, 39 insertions(+), 2 deletions(-)
 create mode 100755 scrub/xfs_scrub_all_fail
 create mode 100644 scrub/xfs_scrub_all_fail.service.in

[PATCHSET 0/5] xfsprogs: file write utility refactoring

[Darrick J. Wong]

Hi all,

Refactor the parts of mkfs and xfs_repair that open-code the process of
mapping disk space into files and writing data into them.  This will help
primarily with resetting of the realtime metadata, but is also used for
protofiles.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=bmap-utils
---
 include/libxfs.h |    6 +-
 libxfs/util.c    |  103 +++++++++++++++++++++++------
 mkfs/proto.c     |  136 ++++++++++----------------------------
 repair/phase6.c  |  194 +++++++++---------------------------------------------
 4 files changed, 154 insertions(+), 285 deletions(-)

[PATCHSET 0/3] xfs_scrub: improve balancing of threads for inode scan

[Darrick J. Wong]

Hi all,

Dave Chinner introduced a patch to the userspace workqueues that allows
for controlling the maximum queue depth as part of phase6 cleanups for
xfs_repair.  This enables us to fix a problem in xfs_scrub wherein we
fail to parallelize inode scans to the maximum extent possible if a
filesystem's inode usage isn't evenly balanced across AGs.

Resolve this problem by using two workqueues for the inode scan -- one
that calls INUMBERS to find all the inobt records for the filesystem and
creates separate work items for each record; and a second workqueue to
turn the inobt records into BULKSTAT calls to do the actual scanning.
We use the queue depth control to avoid excessive queuing of inode
chunks.  This creates more threads to manage, but it means that we avoid
the problem of one AG's inode scan continuing on long after the other
threads ran out of work.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-iscan-rebalance
---
 libfrog/workqueue.c |   42 ++++++
 libfrog/workqueue.h |    4 +
 scrub/inodes.c      |  335 ++++++++++++++++++++++++++++++++++++++-------------
 scrub/scrub.c       |   30 +++++
 scrub/xfs_scrub.c   |    1 
 scrub/xfs_scrub.h   |    3 
 6 files changed, 324 insertions(+), 91 deletions(-)

[PATCHSET 0/4] xfs_scrub: second fixes series

[Darrick J. Wong]

Hi all,

Minor fixes to xfs_scrub the driver program.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=xfs-scrub-fixes
---
 scrub/phase2.c    |    3 ++-
 scrub/phase4.c    |   17 +++++++++++++----
 scrub/scrub.c     |    4 ++++
 scrub/unicrash.c  |   17 +++++++++++++++++
 scrub/unicrash.h  |    4 ++++
 scrub/xfs_scrub.c |    6 ++++++
 6 files changed, 46 insertions(+), 5 deletions(-)

[PATCHSET 00/12] xfs_scrub: track data dependencies for repairs

[Darrick J. Wong]

Hi all,

Certain kinds of XFS metadata depend on the correctness of lower level
metadata.  For example, directory indexes depends on the directory data
fork, which in turn depend on the directoyr inode to be correct.  The
current scrub code does not strictly preserve these dependencies if it
has to defer a repair until phase 4, because phase 4 prioritizes repairs
by type (corruption, then cross referencing, and then preening) and
loses the ordering inherent in the previous phases.

To solve this problem, reorganize the repair ticket to track all the
repairs pending for a princpal object (inode, AG, etc.).  This reduces
memory requirements if an object requires more than one type of repair
and makes it very easy to boost the priority of repairs so that they are
attempted only after dependent data structures are fixed.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-repair-data-deps
---
 scrub/phase1.c |    9 -
 scrub/phase2.c |   53 +++----
 scrub/phase3.c |   49 ++----
 scrub/phase4.c |   10 -
 scrub/phase7.c |    9 -
 scrub/repair.c |  456 +++++++++++++++++++++++++++++++++++++-------------------
 scrub/repair.h |   23 ++-
 scrub/scrub.c  |  288 +++++++++++------------------------
 scrub/scrub.h  |  135 ++++++++++++-----
 9 files changed, 564 insertions(+), 468 deletions(-)

[PATCHSET 0/5] xfstests: establish baseline for fuzz tests

[Darrick J. Wong]

Hi all,

Establish a baseline golden output for the fuzz tests.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzz-baseline
---
 tests/xfs/1554     |   51 ++++
 tests/xfs/1554.out |  392 +++++++++++++++++++++++++++++++
 tests/xfs/1555     |   51 ++++
 tests/xfs/1555.out |   41 +++
 tests/xfs/1556     |   51 ++++
 tests/xfs/1556.out |   10 +
 tests/xfs/1557     |   51 ++++
 tests/xfs/1557.out |   89 +++++++
 tests/xfs/1558     |   51 ++++
 tests/xfs/1558.out |   18 +
 tests/xfs/1559     |   51 ++++
 tests/xfs/1559.out |   10 +
 tests/xfs/350.out  |  133 ++++++++++
 tests/xfs/351.out  |   68 +++++
 tests/xfs/353.out  |  114 +++++++++
 tests/xfs/354.out  |   93 +++++++
 tests/xfs/355.out  |   49 ++++
 tests/xfs/356.out  |   13 +
 tests/xfs/357.out  |  131 ++++++++++
 tests/xfs/358.out  |    5 
 tests/xfs/360.out  |   30 ++
 tests/xfs/361.out  |   14 +
 tests/xfs/362.out  |    5 
 tests/xfs/364.out  |    2 
 tests/xfs/366.out  |    2 
 tests/xfs/368.out  |   11 +
 tests/xfs/369.out  |  190 +++++++++++++++
 tests/xfs/370.out  |  330 ++++++++++++++++++++++++++
 tests/xfs/372.out  |    6 
 tests/xfs/373.out  |   75 ++++++
 tests/xfs/374.out  |  114 +++++++++
 tests/xfs/375.out  |  217 +++++++++++++++++
 tests/xfs/376.out  |  104 ++++++++
 tests/xfs/377.out  |  204 ++++++++++++++++
 tests/xfs/378.out  |  101 ++++++++
 tests/xfs/379.out  |  195 +++++++++++++++
 tests/xfs/380.out  |    6 
 tests/xfs/382.out  |    4 
 tests/xfs/383.out  |    4 
 tests/xfs/384.out  |  122 ++++++++++
 tests/xfs/385.out  |  638 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/386.out  |   49 ++++
 tests/xfs/387.out  |  658 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/388.out  |  369 +++++++++++++++++++++++++++++
 tests/xfs/389.out  |  661 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/392.out  |   18 +
 tests/xfs/393.out  |    5 
 tests/xfs/394.out  |   12 +
 tests/xfs/398.out  |  158 ++++++++++++
 tests/xfs/399.out  |  242 +++++++++++++++++++
 tests/xfs/400.out  |  226 ++++++++++++++++++
 tests/xfs/401.out  |  275 ++++++++++++++++++++++
 tests/xfs/402.out  |    7 +
 tests/xfs/404.out  |   33 +++
 tests/xfs/405.out  |    5 
 tests/xfs/406.out  |   34 +++
 tests/xfs/407.out  |   28 ++
 tests/xfs/408.out  |  143 +++++++++++
 tests/xfs/409.out  |   16 +
 tests/xfs/410.out  |    6 
 tests/xfs/412.out  |  100 ++++++++
 tests/xfs/413.out  |  173 ++++++++++++++
 tests/xfs/414.out  |   98 ++++++++
 tests/xfs/415.out  |  238 +++++++++++++++++++
 tests/xfs/416.out  |   91 +++++++
 tests/xfs/417.out  |  233 ++++++++++++++++++
 tests/xfs/418.out  |  143 +++++++++++
 tests/xfs/425.out  |   90 +++++++
 tests/xfs/426.out  |   65 +++++
 tests/xfs/427.out  |   90 +++++++
 tests/xfs/428.out  |   65 +++++
 tests/xfs/429.out  |   90 +++++++
 tests/xfs/430.out  |   65 +++++
 tests/xfs/453.out  |  169 +++++++++++++
 tests/xfs/454.out  |   99 ++++++++
 tests/xfs/455.out  |  135 +++++++++++
 tests/xfs/456.out  |  132 ++++++++++
 tests/xfs/457.out  |    5 
 tests/xfs/458.out  |   44 +++
 tests/xfs/459.out  |    5 
 tests/xfs/460.out  |    9 +
 tests/xfs/461.out  |    2 
 tests/xfs/462.out  |   12 +
 tests/xfs/463.out  |  351 ++++++++++++++++++++++++++++
 tests/xfs/464.out  |   81 ++++++
 tests/xfs/465.out  |  267 +++++++++++++++++++++
 tests/xfs/466.out  |  249 ++++++++++++++++++++
 tests/xfs/467.out  |  242 +++++++++++++++++++
 tests/xfs/469.out  |    8 +
 tests/xfs/470.out  |  287 +++++++++++++++++++++++
 tests/xfs/471.out  |   22 ++
 tests/xfs/472.out  |   23 ++
 tests/xfs/474.out  |   20 ++
 tests/xfs/475.out  |    6 
 tests/xfs/477.out  |  337 +++++++++++++++++++++++++++
 tests/xfs/478.out  |  492 +++++++++++++++++++++++++++++++++++++++
 tests/xfs/479.out  |    7 +
 tests/xfs/480.out  |   24 ++
 tests/xfs/481.out  |   32 +++
 tests/xfs/482.out  |  142 +++++++++++
 tests/xfs/483.out  |    6 
 tests/xfs/484.out  |  239 +++++++++++++++++++
 tests/xfs/485.out  |  224 ++++++++++++++++++
 tests/xfs/486.out  |  235 ++++++++++++++++++
 tests/xfs/487.out  |  137 +++++++++++
 tests/xfs/488.out  |  137 +++++++++++
 tests/xfs/489.out  |  137 +++++++++++
 tests/xfs/496.out  |   24 ++
 tests/xfs/498.out  |   12 +
 tests/xfs/713.out  |   14 +
 tests/xfs/group    |    6 
 111 files changed, 12704 insertions(+), 1 deletion(-)
 create mode 100755 tests/xfs/1554
 create mode 100644 tests/xfs/1554.out
 create mode 100755 tests/xfs/1555
 create mode 100644 tests/xfs/1555.out
 create mode 100755 tests/xfs/1556
 create mode 100644 tests/xfs/1556.out
 create mode 100755 tests/xfs/1557
 create mode 100644 tests/xfs/1557.out
 create mode 100755 tests/xfs/1558
 create mode 100644 tests/xfs/1558.out
 create mode 100755 tests/xfs/1559
 create mode 100644 tests/xfs/1559.out

[PATCHSET 00/15] xfstests: improve xfs fuzzing

[Darrick J. Wong]

Hi all,

There are a ton of improvements to the XFS fuzzing code in this update.
We start by disabling by default two parts of the fuzz testing that
don't lead to predictable golden output: fuzzing with the 'random' verb,
and fuzzing the 'LSN' field.

Next, we refactor the inner fuzzing loop so that each of the four repair
strategies are broken out into separate functions, as well as the
post-repair attempts to modify the filesystem.

The next five patches after that modify each of the functions that we
split out in the previous patches to make what those functions do much
more clear.  We also revise the strategies a bit so that they more
accurately reflect the intended usage patterns of the repair programs.

Next, we strengthen other parts of the fuzzing -- we make the
post-repair modification exercises a bit more strenuous, add an
evaluation of xfs_check vs. xfs_repair, and make it possible to check
the xfs health reporting system.

Finally, we improve the array handling of the xfs fuzz tests so that we
actually know about array indices as a first class concept, instead of
the current mucking around we do with regular expressions.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzzer-improvements
---
 common/fuzzy  |  461 +++++++++++++++++++++++++++++++++++++++++++--------------
 common/xfs    |   38 +++++
 tests/xfs/357 |    2 
 3 files changed, 386 insertions(+), 115 deletions(-)

[PATCHSET 0/2] xfstests: strengthen fuzz testing

[Darrick J. Wong]

Hi all,

Add new fuzz tests to try the "online then offline" repair strategy.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=more-fuzz-testing
---
 tests/xfs/1500     |   46 +++++
 tests/xfs/1500.out |  173 ++++++++++++++++++
 tests/xfs/1501     |   46 +++++
 tests/xfs/1501.out |  117 ++++++++++++
 tests/xfs/1502     |   51 +++++
 tests/xfs/1502.out |  148 +++++++++++++++
 tests/xfs/1503     |   46 +++++
 tests/xfs/1503.out |  156 ++++++++++++++++
 tests/xfs/1504     |   46 +++++
 tests/xfs/1504.out |    9 +
 tests/xfs/1505     |   46 +++++
 tests/xfs/1505.out |   48 +++++
 tests/xfs/1506     |   46 +++++
 tests/xfs/1506.out |    9 +
 tests/xfs/1507     |   46 +++++
 tests/xfs/1507.out |    4 
 tests/xfs/1508     |   47 +++++
 tests/xfs/1508.out |    6 +
 tests/xfs/1509     |   47 +++++
 tests/xfs/1509.out |  254 ++++++++++++++++++++++++++
 tests/xfs/1510     |   47 +++++
 tests/xfs/1510.out |  355 +++++++++++++++++++++++++++++++++++++
 tests/xfs/1511     |   48 +++++
 tests/xfs/1511.out |   85 +++++++++
 tests/xfs/1512     |   51 +++++
 tests/xfs/1512.out |  309 ++++++++++++++++++++++++++++++++
 tests/xfs/1513     |   51 +++++
 tests/xfs/1513.out |  301 +++++++++++++++++++++++++++++++
 tests/xfs/1514     |   51 +++++
 tests/xfs/1514.out |  289 ++++++++++++++++++++++++++++++
 tests/xfs/1515     |   53 ++++++
 tests/xfs/1515.out |    5 +
 tests/xfs/1516     |   51 +++++
 tests/xfs/1516.out |   13 +
 tests/xfs/1517     |   51 +++++
 tests/xfs/1517.out |  416 +++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1518     |   51 +++++
 tests/xfs/1518.out |  179 +++++++++++++++++++
 tests/xfs/1519     |   52 +++++
 tests/xfs/1519.out |  262 +++++++++++++++++++++++++++
 tests/xfs/1520     |   53 ++++++
 tests/xfs/1520.out |    5 +
 tests/xfs/1521     |   53 ++++++
 tests/xfs/1521.out |   25 +++
 tests/xfs/1522     |   53 ++++++
 tests/xfs/1522.out |   11 +
 tests/xfs/1523     |   53 ++++++
 tests/xfs/1523.out |    5 +
 tests/xfs/1524     |   51 +++++
 tests/xfs/1524.out |  381 ++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1525     |   51 +++++
 tests/xfs/1525.out |  501 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1526     |   51 +++++
 tests/xfs/1526.out |   12 +
 tests/xfs/1527     |   51 +++++
 tests/xfs/1527.out |   29 +++
 tests/xfs/1528     |   50 +++++
 tests/xfs/1528.out |   20 ++
 tests/xfs/1529     |   48 +++++
 tests/xfs/1529.out |  146 +++++++++++++++
 tests/xfs/1530     |   48 +++++
 tests/xfs/1530.out |   10 +
 tests/xfs/1531     |   51 +++++
 tests/xfs/1531.out |  271 ++++++++++++++++++++++++++++
 tests/xfs/1532     |   51 +++++
 tests/xfs/1532.out |  291 ++++++++++++++++++++++++++++++
 tests/xfs/1533     |   51 +++++
 tests/xfs/1533.out |  300 +++++++++++++++++++++++++++++++
 tests/xfs/1534     |   49 +++++
 tests/xfs/1534.out |  141 +++++++++++++++
 tests/xfs/1535     |   49 +++++
 tests/xfs/1535.out |  141 +++++++++++++++
 tests/xfs/1536     |   49 +++++
 tests/xfs/1536.out |  141 +++++++++++++++
 tests/xfs/1537     |   52 +++++
 tests/xfs/1537.out |   17 ++
 tests/xfs/1560     |   52 +++++
 tests/xfs/1560.out |  220 +++++++++++++++++++++++
 tests/xfs/1561     |   52 +++++
 tests/xfs/1561.out |   20 ++
 tests/xfs/group    |   40 ++++
 81 files changed, 7856 insertions(+)
 create mode 100755 tests/xfs/1500
 create mode 100644 tests/xfs/1500.out
 create mode 100755 tests/xfs/1501
 create mode 100644 tests/xfs/1501.out
 create mode 100755 tests/xfs/1502
 create mode 100644 tests/xfs/1502.out
 create mode 100755 tests/xfs/1503
 create mode 100644 tests/xfs/1503.out
 create mode 100755 tests/xfs/1504
 create mode 100644 tests/xfs/1504.out
 create mode 100755 tests/xfs/1505
 create mode 100644 tests/xfs/1505.out
 create mode 100755 tests/xfs/1506
 create mode 100644 tests/xfs/1506.out
 create mode 100755 tests/xfs/1507
 create mode 100644 tests/xfs/1507.out
 create mode 100755 tests/xfs/1508
 create mode 100644 tests/xfs/1508.out
 create mode 100755 tests/xfs/1509
 create mode 100644 tests/xfs/1509.out
 create mode 100755 tests/xfs/1510
 create mode 100644 tests/xfs/1510.out
 create mode 100755 tests/xfs/1511
 create mode 100644 tests/xfs/1511.out
 create mode 100755 tests/xfs/1512
 create mode 100644 tests/xfs/1512.out
 create mode 100755 tests/xfs/1513
 create mode 100644 tests/xfs/1513.out
 create mode 100755 tests/xfs/1514
 create mode 100644 tests/xfs/1514.out
 create mode 100755 tests/xfs/1515
 create mode 100644 tests/xfs/1515.out
 create mode 100755 tests/xfs/1516
 create mode 100644 tests/xfs/1516.out
 create mode 100755 tests/xfs/1517
 create mode 100644 tests/xfs/1517.out
 create mode 100755 tests/xfs/1518
 create mode 100644 tests/xfs/1518.out
 create mode 100755 tests/xfs/1519
 create mode 100644 tests/xfs/1519.out
 create mode 100755 tests/xfs/1520
 create mode 100644 tests/xfs/1520.out
 create mode 100755 tests/xfs/1521
 create mode 100644 tests/xfs/1521.out
 create mode 100755 tests/xfs/1522
 create mode 100644 tests/xfs/1522.out
 create mode 100755 tests/xfs/1523
 create mode 100644 tests/xfs/1523.out
 create mode 100755 tests/xfs/1524
 create mode 100644 tests/xfs/1524.out
 create mode 100755 tests/xfs/1525
 create mode 100644 tests/xfs/1525.out
 create mode 100755 tests/xfs/1526
 create mode 100644 tests/xfs/1526.out
 create mode 100755 tests/xfs/1527
 create mode 100644 tests/xfs/1527.out
 create mode 100755 tests/xfs/1528
 create mode 100644 tests/xfs/1528.out
 create mode 100755 tests/xfs/1529
 create mode 100644 tests/xfs/1529.out
 create mode 100755 tests/xfs/1530
 create mode 100644 tests/xfs/1530.out
 create mode 100755 tests/xfs/1531
 create mode 100644 tests/xfs/1531.out
 create mode 100755 tests/xfs/1532
 create mode 100644 tests/xfs/1532.out
 create mode 100755 tests/xfs/1533
 create mode 100644 tests/xfs/1533.out
 create mode 100755 tests/xfs/1534
 create mode 100644 tests/xfs/1534.out
 create mode 100755 tests/xfs/1535
 create mode 100644 tests/xfs/1535.out
 create mode 100755 tests/xfs/1536
 create mode 100644 tests/xfs/1536.out
 create mode 100755 tests/xfs/1537
 create mode 100644 tests/xfs/1537.out
 create mode 100755 tests/xfs/1560
 create mode 100644 tests/xfs/1560.out
 create mode 100755 tests/xfs/1561
 create mode 100644 tests/xfs/1561.out

[PATCHSET 0/3] common/dm*: support external log and rt devices

[Darrick J. Wong]

Hi all,

There are a growing number of fstests that examine what happens to a
filesystem when the block device underneath it goes offline.  Many of
them do this to simulate system crashes, and none of them (outside of
btrfs) can handle filesystems with multiple devices.  XFS is one of
those beasts that does, so enhance the dm-error and dm-flakey helpers to
take the log and rt devices offline when they're testing the data
device.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=dmerror-on-rt-devices
---
 common/dmerror    |  163 ++++++++++++++++++++++++++++++++++++++++++++++++++---
 common/dmflakey   |  106 +++++++++++++++++++++++++++++++++-
 tests/generic/250 |    3 -
 tests/generic/252 |    3 -
 tests/generic/441 |    3 -
 tests/generic/484 |    3 -
 tests/generic/487 |    3 -
 7 files changed, 257 insertions(+), 27 deletions(-)

[PATCHSET 0/3] common/dm*: support external log and rt devices

[Darrick J. Wong]

Hi all,

There are a growing number of fstests that examine what happens to a
filesystem when the block device underneath it goes offline.  Many of
them do this to simulate system crashes, and none of them (outside of
btrfs) can handle filesystems with multiple devices.  XFS is one of
those beasts that does, so enhance the dm-error and dm-flakey helpers to
take the log and rt devices offline when they're testing the data
device.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=dmerror-on-rt-devices
---
 common/dmerror    |  190 ++++++++++++++++++++++++++++++++++++++++++++++++++---
 common/dmflakey   |  106 +++++++++++++++++++++++++++++-
 src/dmerror       |   13 +---
 tests/generic/441 |    2 -
 tests/generic/487 |    2 -
 5 files changed, 289 insertions(+), 24 deletions(-)