fstests.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Darrick J. Wong" <djwong@kernel.org>
To: zlang@redhat.com, djwong@kernel.org
Cc: linux-xfs@vger.kernel.org, fstests@vger.kernel.org, guan@eryu.me
Subject: [PATCHSET v24.0 0/2] fstests: online repair of directories
Date: Fri, 30 Dec 2022 14:19:57 -0800	[thread overview]
Message-ID: <167243879781.733381.1441585366549762189.stgit@magnolia> (raw)
In-Reply-To: <Y69Unb7KRM5awJoV@magnolia>

Hi all,

This series employs atomic extent swapping to enable safe reconstruction
of directory data.  For now, XFS does not support reverse directory
links (aka parent pointers), so we can only salvage the dirents of a
directory and construct a new structure.

Directory repair therefore consists of five main parts:

First, we walk the existing directory to salvage as many entries as we
can, by adding them as new directory entries to the repair temp dir.

Second, we validate the parent pointer found in the directory.  If one
was not found, we scan the entire filesystem looking for a potential
parent.

Third, we use atomic extent swaps to exchange the entire data fork
between the two directories.

Fourth, we reap the old directory blocks as carefully as we can.

To wrap up the directory repair code, we need to add to the regular
filesystem the ability to free all the data fork blocks in a directory.
This does not change anything with normal directories, since they must
still unlink and shrink one entry at a time.  However, this will
facilitate freeing of partially-inactivated temporary directories during
log recovery.

The second half of this patchset implements repairs for the dotdot
entries of directories.  For now there is only rudimentary support for
this, because there are no directory parent pointers, so the best we can
do is scanning the filesystem and the VFS dcache for answers.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-dirs

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-dirs
---
 tests/xfs/815     |   37 +++++++++++++++++++++++++
 tests/xfs/815.out |    2 +
 tests/xfs/816     |   38 ++++++++++++++++++++++++++
 tests/xfs/816.out |    2 +
 tests/xfs/841     |   77 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/841.out |    3 ++
 6 files changed, 159 insertions(+)
 create mode 100755 tests/xfs/815
 create mode 100644 tests/xfs/815.out
 create mode 100755 tests/xfs/816
 create mode 100644 tests/xfs/816.out
 create mode 100755 tests/xfs/841
 create mode 100644 tests/xfs/841.out


  parent reply	other threads:[~2022-12-30 23:19 UTC|newest]

Thread overview: 106+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong
2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong
2022-12-30 22:19   ` [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata Darrick J. Wong
2023-02-05 13:04     ` Zorro Lang
2023-02-07 16:58       ` Darrick J. Wong
2023-02-07 17:02     ` [PATCH v24.1 " Darrick J. Wong
2023-02-07 18:45       ` Zorro Lang
2022-12-30 22:19   ` [PATCH 5/5] xfs: race fsstress with online scrubbers for file metadata Darrick J. Wong
2022-12-30 22:19   ` [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests Darrick J. Wong
2023-02-05 12:57     ` Zorro Lang
2023-02-07 16:57       ` Darrick J. Wong
2023-02-07 17:01     ` [PATCH v24.1 " Darrick J. Wong
2023-02-07 18:42       ` Zorro Lang
2022-12-30 22:19   ` [PATCH 1/5] xfs/357: switch fuzzing to agi 1 Darrick J. Wong
2023-02-07 18:46     ` Zorro Lang
2022-12-30 22:19   ` [PATCH 4/5] fuzzy: allow xfs scrub stress tests to pick preconfigured fsstress configs Darrick J. Wong
2023-02-07 18:48     ` Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/1] xfs: force rebuilding of metadata Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/1] fuzzy: use FORCE_REBUILD over injecting force_repair Darrick J. Wong
2023-02-14  8:00     ` Zorro Lang
2023-02-14 18:18       ` Darrick J. Wong
2023-02-16 14:57         ` Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Darrick J. Wong
2022-12-30 22:19   ` [PATCH 2/2] xfs: stress test ag repair functions Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/2] xfs: test rebuilding the entire filesystem with online fsck Darrick J. Wong
2023-02-18  6:06   ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of inodes Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/1] xfs: race fsstress with online repair for inode record metadata Darrick J. Wong
2023-02-18  6:07     ` Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong
2022-12-30 22:19   ` [PATCH 4/4] xfs: race fsstress with online repair for special file metadata Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/4] xfs: test rebuilding xattrs when the data fork is btree format Darrick J. Wong
2022-12-30 22:19   ` [PATCH 3/4] xfs: ensure that online file data fork repairs don't hit EDQUOT Darrick J. Wong
2022-12-30 22:19   ` [PATCH 2/4] xfs: race fsstress with online repair for inode and fork metadata Darrick J. Wong
2023-02-18  6:07   ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota and counters Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quota metadata Darrick J. Wong
2023-02-18  6:10     ` Zorro Lang
2023-02-18  6:12     ` Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota counters Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quotacheck Darrick J. Wong
2023-02-18  6:12     ` Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of file link counts Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/1] xfs: race fsstress with inode link count check and repair Darrick J. Wong
2023-02-18  6:13     ` Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/2] xfs: test fs summary counter online repair Darrick J. Wong
2022-12-30 22:19   ` [PATCH 2/2] xfs: race fsstress with online repair for summary counters Darrick J. Wong
2023-02-18  6:14   ` [PATCHSET v24.0 0/2] fstests: online repair for fs " Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of rmap btrees Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/1] xfs/422: don't freeze while racing rmap repair and fsstress Darrick J. Wong
2023-02-18  6:15     ` Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/2] populate: take a snapshot of the filesystem if creation fails Darrick J. Wong
2022-12-30 22:19   ` [PATCH 2/2] populate: fix some weirdness in __populate_check_xfs_agbtree_height Darrick J. Wong
2023-02-18  6:16   ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Zorro Lang
2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong
2022-12-30 22:19   ` [PATCH 01/24] fuzzy: disable per-field random fuzzing by default Darrick J. Wong
2022-12-30 22:19   ` [PATCH 02/24] fuzzy: disable timstamp " Darrick J. Wong
2022-12-30 22:19   ` [PATCH 07/24] fuzzy: don't fuzz xattr namespace flags and values Darrick J. Wong
2022-12-30 22:19   ` [PATCH 05/24] fuzzy: don't fuzz inode generation numbers Darrick J. Wong
2022-12-30 22:19   ` [PATCH 03/24] fuzzy: don't fuzz the log sequence number Darrick J. Wong
2022-12-30 22:19   ` [PATCH 06/24] fuzzy: don't fuzz user-controllable inode flags Darrick J. Wong
2022-12-30 22:19   ` [PATCH 04/24] fuzzy: don't fuzz obsolete inode fields Darrick J. Wong
2022-12-30 22:19   ` [PATCH 12/24] common/fuzzy: fix some problems with the offline repair strategy Darrick J. Wong
2022-12-30 22:19   ` [PATCH 11/24] common/fuzzy: fix some problems with the online " Darrick J. Wong
2022-12-30 22:19   ` [PATCH 08/24] common/fuzzy: split out each repair strategy into a separate helper Darrick J. Wong
2022-12-30 22:19   ` [PATCH 10/24] common/fuzzy: hoist the post-repair fs modification step Darrick J. Wong
2022-12-30 22:19   ` [PATCH 09/24] common/fuzzy: add an underline to the full log between sections Darrick J. Wong
2022-12-30 22:19   ` [PATCH 13/24] common/fuzzy: fix some problems with the no-repair strategy Darrick J. Wong
2022-12-30 22:19   ` [PATCH 14/24] common/fuzzy: fix some problems with the online-then-offline repair strategy Darrick J. Wong
2022-12-30 22:19   ` [PATCH 16/24] xfs/{35[45],455}: fix bogus corruption errors Darrick J. Wong
2022-12-30 22:19   ` [PATCH 21/24] fuzzy: compress coredumps created while fuzzing Darrick J. Wong
2022-12-30 22:19   ` [PATCH 15/24] common/fuzzy: fix some problems with the post-repair fs modification code Darrick J. Wong
2022-12-30 22:19   ` [PATCH 19/24] common/fuzzy: exercise the filesystem a little harder after repairing Darrick J. Wong
2022-12-30 22:19   ` [PATCH 22/24] fuzzy: report the fuzzing repair strategy in seqres.full Darrick J. Wong
2022-12-30 22:19   ` [PATCH 17/24] common/fuzzy: evaluate xfs_check vs xfs_repair Darrick J. Wong
2022-12-30 22:19   ` [PATCH 20/24] fuzzy: dump metadata state before fuzzing Darrick J. Wong
2022-12-30 22:19   ` [PATCH 18/24] common: check xfs health after doing an online scrub Darrick J. Wong
2022-12-30 22:19   ` [PATCH 24/24] fuzzy: for fuzzing ag btrees, find the path to the AG header Darrick J. Wong
2022-12-30 22:19   ` [PATCH 23/24] xfs: improve metadata array field handling when fuzzing Darrick J. Wong
2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/5] fuzzy: test fuzzing directory block mappings Darrick J. Wong
2022-12-30 22:19   ` [PATCH 2/5] fuzzy: test fuzzing xattr " Darrick J. Wong
2022-12-30 22:19   ` [PATCH 5/5] fuzzy: fuzz test key/pointers of inode btrees Darrick J. Wong
2022-12-30 22:19   ` [PATCH 4/5] xfs: fuzz test both repair strategies Darrick J. Wong
2022-12-30 22:19   ` [PATCH 3/5] fuzzy: test fuzzing realtime free space metadata Darrick J. Wong
2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/7] xfs/122: fix for swapext log items Darrick J. Wong
2022-12-30 22:19   ` [PATCH 5/7] generic: test that file privilege gets dropped with FIEXCHANGE_RANGE Darrick J. Wong
2022-12-30 22:19   ` [PATCH 4/7] generic, xfs: test scatter-gather atomic file updates Darrick J. Wong
2022-12-30 22:19   ` [PATCH 3/7] generic: test new vfs swapext ioctl Darrick J. Wong
2022-12-30 22:19   ` [PATCH 2/7] generic: test old xfs extent swapping ioctl Darrick J. Wong
2022-12-30 22:19   ` [PATCH 7/7] fsstress: update for FIEXCHANGE_RANGE Darrick J. Wong
2022-12-30 22:19   ` [PATCH 6/7] fsx: support FIEXCHANGE_RANGE Darrick J. Wong
2023-02-28  1:55     ` Zorro Lang
2023-03-01  2:56       ` Darrick J. Wong
2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of realtime summaries Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/1] xfs: race fsstress with online repair of realtime summary files Darrick J. Wong
2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of extended attributes Darrick J. Wong
2022-12-30 22:19   ` [PATCH 1/1] xfs: race fsstress with online repair of extended attribute data Darrick J. Wong
2022-12-30 22:19 ` Darrick J. Wong [this message]
2022-12-30 22:19   ` [PATCH 1/2] xfs: ensure that online directory repairs don't hit EDQUOT Darrick J. Wong
2022-12-30 22:19   ` [PATCH 2/2] xfs: race fsstress with online repair of dirs and parent pointers Darrick J. Wong
2022-12-30 22:20 ` [PATCHSET v24.0 0/1] fstests: test automatic scrub optimization by default Darrick J. Wong
2022-12-30 22:20   ` [PATCH 1/1] xfs: test xfs_scrub dry run, preen, and repair mode Darrick J. Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=167243879781.733381.1441585366549762189.stgit@magnolia \
    --to=djwong@kernel.org \
    --cc=fstests@vger.kernel.org \
    --cc=guan@eryu.me \
    --cc=linux-xfs@vger.kernel.org \
    --cc=zlang@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).