2019 NYE Patchbomb Guide!

2019 NYE Patchbomb Guide!

[Darrick Wong]

Hi folks,

It's still the last day of 2019 in the US/Pacific timezone, which means
it's time for me to patchbomb every new feature that's been sitting
around in my development trees!  As you know, all of my development
branches are kept freshly rebased on the latest Linus kernel so that I
can send them all at a moment's notice.  Once a year I like to dump
everything into the public archives to increase XFS' bus factor.

To make things easier for everyone, I'm sending this short note to help
readers figure out how each of the major branches depend on each other.
The branch mentioned on any line within one of the small lists depends
on the branches mentioned in the previous lines.  Separation between
lists indicate lack of a hard dependency between branches, except as
noted.  For the kernel, this looks like:

btree-bulk-loading		("xfs: btree bulk loading")
repair-reap-fixes		("xfs: fix online repair block reaping")
repair-bitmap-rework		("xfs: rework online repair incore bitmap")
repair-prep-for-bulk-loading	("xfs: prepare repair for bulk loading")

These four series have been out for review before; they're the prep work
needed to land the next four, which implement the easier parts of online
fsck:

repair-ag-btrees		("xfs: online repair of AG btrees")
repair-inodes			("xfs: online repair of inodes and extent maps")
repair-inode-data		("xfs: online repair of inode data")
repair-quota			("xfs: online repair of quota")

This next series ties corruption reporting to the health reporting
subsystem so that we can enable targeted spot repairs of filesystems in
the next phase of xfs_scrub development.  It isn't dependent upon any
previous series (though I think it still needs some more work).

corruption-health-reports	("xfs: report corruption to the health trackers")

We now turn our attention to the longstanding problem that inode
inactivation can require on-disk metadata updates, which are very time
consuming.  This is the root cause of all the complaints on the mailing
list about kernel memory reclamation stalling behind XFS.  There's no
hard dependency between these series and any of the ones before it.

incore-inode-walk		("xfs: refactor incore inode walking")
reclaim-space-harder		("xfs: try harder to reclaim space when we run out")
eofblocks-consolidation		("xfs: consolidate posteof and cowblocks cleanup")
deferred-inactivation		("xfs: deferred inode inactivation")

These next two series tackle some harder repair problems (namely the
ones that require full filesystem scans).  These scans depend upon being
able to shut down background operations temporarily; these two series
actually /do/ depend on the previous four series.

indirect-health-reporting	("xfs: indirect health reporting")
repair-hard-problems		("xfs: online repair of rmap/quota/summary counters")

These next two branches are grouped together because they involve
on-disk format changes.  One will reduce mount time on finobt=1
filesystems; the other aims to guarantee that XFS will continue to work
with timestamps through year 2485.  They have no hard dependencies on
the previous series.

inobt-counters			("xfs: add a inode btree blocks counts to the AGI header")
bigtime				("xfs: widen timestamps to deal with y2038")

These next two series introduce the metadata directory tree, which will
enable us to add new metadata type without filling up more of the
superblock.  They don't rely much on any of the previous series, but
they're not immediately applyable unless you've already applied the
other patches that touch the inode code.

inode-refactor			("xfs: hoist inode operations to libxfs")
metadir				("xfs: metadata inode directories")

This last series uses the metadata directory tree functionality to add
reverse mapping abilities to the realtime volume.  It's the precursor to
supporting reflink and COW on the realtime device.

realtime-rmap			("xfs: realtime reverse-mapping support")

========

For xfsprogs, the series look like this:

libfrog-fixes			("libfrog: various small fixes")
random-fixes			("xfsprogs: random fixes")

Thse two series apply cleanups to xfsprogs.

repair-find-rootdir		("xfs_repair: do not trash valid root dirs")

This series attempts to fix xfs_repair so it doesn't toss the root
directory when someone applied an improper sunit change via mount
option.

btree-bulk-loading		("libxfs: btree bulk loading")
repair-ag-btrees		("libxfs: online repair of AG btrees")
repair-inodes			("libxfs: online repair of inodes and extent maps")
repair-inode-data		("libxfs: online repair of inode data")

This series refactors repair to use the btree bulk loading code from
above.

repair-bulk-load		("xfs_repair: use btree bulk loading")

This series teaches repair to use rmap data to rebuild inode extent
mappings using the btree bulk loading code from above.

repair-rebuild-forks		("xfs_repair: rebuild inode fork mappings")

The rest of these are partly ports of the kernel code and minimal
updates to the existing utilities to enable those new features.

corruption-health-reports	("xfs: report corruption to the health trackers")
indirect-health-reporting	("libxfs: indirect health reporting")
repair-hard-problems		("libxfs: online repair of rmap/quota/summary counters")
inobt-counters			("xfs: add a inode btree blocks counts to the AGI header")
bigtime				("xfs: widen timestamps to deal with y2038")
inode-refactor			("xfsprogs: hoist inode operations to libxfs")
metadir				("xfsprogs: metadata inode directories")
realtime-rmap			("xfsprogs: realtime reverse-mapping support")

Finally, here are a couple more packging cleanups and fixes for
mkfs protofile support that might get dropped later.

packaging-cleanups		("xfsprogs: packaging cleanups")
bmap-utils			("xfsprogs: bmap utility refactoring")

========

For what it's worth, here are a few notable things sitting in my fstests
development tree.  I didn't send any of these to the list.

These three series are tests for things that have either gone in
recently or are nearly in.

scrub-media-error-reporting	("xfs: test xfs_scrub media scan")
online-label-setting		("xfs_admin: unify online/offline fs label setting")
repair-find-rootdir		("xfs_repair: do not trash valid root dirs")

And these three are for the new features mentioned above.

bigtime				("xfs: widen timestamps to deal with y2038")
metadir				("xfs: fixes for metadata directories")
fuzz-baseline			("xfs: establish baseline for fuzz tests")

With that, I hope you all have a happy new year!  See you in 2020/4717!

--D

Re: 2019 NYE Patchbomb Guide!

[Christoph Hellwig]

On Tue, Dec 31, 2019 at 05:25:08PM -0800, Darrick Wong wrote:
> Hi folks,
> 
> It's still the last day of 2019 in the US/Pacific timezone, which means
> it's time for me to patchbomb every new feature that's been sitting
> around in my development trees!  As you know, all of my development
> branches are kept freshly rebased on the latest Linus kernel so that I
> can send them all at a moment's notice.  Once a year I like to dump
> everything into the public archives to increase XFS' bus factor.

It seems like you missed the stale data exposure fix series using
unwritten extents in buffered writeback.  Can we get that one off the
back burner?

Re: 2019 NYE Patchbomb Guide!

[Darrick J. Wong]

On Wed, Jan 15, 2020 at 07:12:19AM -0800, Christoph Hellwig wrote:
> On Tue, Dec 31, 2019 at 05:25:08PM -0800, Darrick Wong wrote:
> > Hi folks,
> > 
> > It's still the last day of 2019 in the US/Pacific timezone, which means
> > it's time for me to patchbomb every new feature that's been sitting
> > around in my development trees!  As you know, all of my development
> > branches are kept freshly rebased on the latest Linus kernel so that I
> > can send them all at a moment's notice.  Once a year I like to dump
> > everything into the public archives to increase XFS' bus factor.
> 
> It seems like you missed the stale data exposure fix series using
> unwritten extents in buffered writeback.  Can we get that one off the
> back burner?

Aha, at least two people read the guide! :)

Yeah, I'll repost them shortly, though iirc the last time I posted them
Dave complained about the performance impacts.

Personally I think it's more important to guarantee that we can never
return disk garbage after a crash, but nobody RVBd the series so it
stalled.

--D