Re: [PATCH 5/5] xfs: add online scrub for superblock counters

From: Dave Chinner <david@fromorbit.com>
To: "Darrick J. Wong" <darrick.wong@oracle.com>
Cc: linux-xfs@vger.kernel.org
Subject: Re: [PATCH 5/5] xfs: add online scrub for superblock counters
Date: Fri, 26 Apr 2019 09:30:59 +1000	[thread overview]
Message-ID: <20190425233059.GF29573@dread.disaster.area> (raw)
In-Reply-To: <155563328923.112668.18007995408695103782.stgit@magnolia>

On Thu, Apr 18, 2019 at 05:21:29PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Teach online scrub how to check the filesystem summary counters.  We use
> the incore delalloc block counter along with the incore AG headers to
> compute expected values for fdblocks, icount, and ifree, and then check
> that the percpu counter is within a certain threshold of the expected
> value.  This is done to avoid having to freeze or otherwise lock the
> filesystem, which means that we're only checking that the counters are
> fairly close, not that they're exactly correct.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
....
> --- /dev/null
> +++ b/fs/xfs/scrub/fscounters.c
> @@ -0,0 +1,348 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Copyright (C) 2019 Oracle.  All Rights Reserved.
> + * Author: Darrick J. Wong <darrick.wong@oracle.com>
> + */
> +#include "xfs.h"
> +#include "xfs_fs.h"
> +#include "xfs_shared.h"
> +#include "xfs_format.h"
> +#include "xfs_trans_resv.h"
> +#include "xfs_mount.h"
> +#include "xfs_defer.h"
> +#include "xfs_btree.h"
> +#include "xfs_bit.h"
> +#include "xfs_log_format.h"
> +#include "xfs_trans.h"
> +#include "xfs_sb.h"
> +#include "xfs_inode.h"
> +#include "xfs_alloc.h"
> +#include "xfs_ialloc.h"
> +#include "xfs_rmap.h"
> +#include "xfs_error.h"
> +#include "xfs_errortag.h"
> +#include "xfs_icache.h"
> +#include "xfs_health.h"
> +#include "xfs_bmap.h"
> +#include "scrub/xfs_scrub.h"
> +#include "scrub/scrub.h"
> +#include "scrub/common.h"
> +#include "scrub/trace.h"
> +
> +/*
> + * FS Summary Counters
> + * ===================
> + *
> + * The basics of filesystem summary counter checking are that we iterate the
> + * AGs counting the number of free blocks, free space btree blocks, per-AG
> + * reservations, inodes, delayed allocation reservations, and free inodes.
> + * Then we compare what we computed against the in-core counters.
> + *
> + * However, the reality is that summary counters are a tricky beast to check.
> + * While we /could/ freeze the filesystem and scramble around the AGs counting
> + * the free blocks, in practice we prefer not do that for a scan because
> + * freezing is costly.  To get around this, we added a per-cpu counter of the
> + * delalloc reservations so that we can rotor around the AGs relatively
> + * quickly, and we allow the counts to be slightly off because we're not
> + * taking any locks while we do this.
> + *
> + * So the first thing we do is warm up the buffer cache in the setup routine
> + * by walking all the AGs to make sure the incore per-AG structure has been
> + * initialized.  The expected value calculation then iterates the incore per-AG
> + * structures as quickly as it can.  We snapshot the percpu counters before and
> + * after this operation and use the difference in counter values to guess at
> + * our tolerance for mismatch between expected and actual counter values.
> + */
> +
> +/*
> + * Since the expected value computation is lockless but only browses incore
> + * values, the percpu counters should be fairly close to each other.  However,
> + * we'll allow ourselves to be off by at least this (arbitrary) amount.
> + */
> +#define XCHK_FSC_MIN_VARIANCE	(512)
> +
> +/*
> + * Make sure the per-AG structure has been initialized from the on-disk header
> + * contents and that the incore counters match the ondisk counters.  Do this
> + * from the setup function so that the inner summation loop runs as quickly
> + * as possible.
> + */
> +STATIC int
> +xchk_fsc_warmup(
> +	struct xfs_scrub	*sc)
> +{
> +	struct xfs_mount	*mp = sc->mp;
> +	struct xfs_buf		*agi_bp = NULL;
> +	struct xfs_buf		*agf_bp = NULL;
> +	struct xfs_agi		*agi;
> +	struct xfs_agf		*agf;
> +	struct xfs_perag	*pag = NULL;
> +	xfs_agnumber_t		agno;
> +	int			error = 0;
> +
> +	for (agno = 0; agno < mp->m_sb.sb_agcount; agno++) {
> +		/* Lock both AG headers. */
> +		error = xfs_ialloc_read_agi(mp, sc->tp, agno, &agi_bp);
> +		if (error)
> +			break;
> +		error = xfs_alloc_read_agf(mp, sc->tp, agno, 0, &agf_bp);
> +		if (error)
> +			break;
> +		error = -ENOMEM;
> +		if (!agf_bp)
> +			break;
> +
> +		pag = xfs_perag_get(mp, agno);
> +
> +		/*
> +		 * These are supposed to be initialized by the header read
> +		 * function.
> +		 */
> +		error = -EFSCORRUPTED;
> +		if (!pag->pagi_init || !pag->pagf_init)
> +			break;
> +
> +		/* Spot-check the incore counters against the ondisk headers */
> +		agi = XFS_BUF_TO_AGI(agi_bp);
> +		agf = XFS_BUF_TO_AGF(agf_bp);
> +		if (pag->pagi_count != be32_to_cpu(agi->agi_count))
> +			break;
> +		if (pag->pagi_freecount != be32_to_cpu(agi->agi_freecount))
> +			break;
> +		if (pag->pagf_freeblks != be32_to_cpu(agf->agf_freeblks))
> +			break;
> +		if (pag->pagf_flcount != be32_to_cpu(agf->agf_flcount))
> +			break;
> +		if (pag->pagf_btreeblks != be32_to_cpu(agf->agf_btreeblks))
> +			break;
> +
> +		xfs_perag_put(pag);
> +		pag = NULL;
> +		xfs_buf_relse(agf_bp);
> +		agf_bp = NULL;
> +		xfs_buf_relse(agi_bp);
> +		agi_bp = NULL;
> +		error = 0;
> +	}

This will work, but it's going to be expensive on a high AG count
filesystem (2 synchronous reads per AG). On multi-PB filesystems
under memory pressure, this could require tens of thousands of IOs
to be done, and on slow/overloaded disks this could take tens of
minutes or even hours just to do this initial check.

Perhaps we should start thinking about this sort of loop as an async
read with a special verifier to check the pag values and store the
result in the pag? After all, the pag will be attached to the buffer
and available at IO completion, and this way we can just rattle off
IO submission then when we do the incore pag walk we can look for
the "check good/bad" flag in the pag and only wait if neither are
set?

This will allow the AGF/AGI IOs to be merged at the block layer
(immediate 50% reduction in IO), and we'll only get throttled on
submission when the queue gets too long. This will greatly reduce
the "warmup" pre-check time, especially if the disk is loaded and
individual IO times are measured in tens to hundreds of
milliseconds...

Perhaps this can be done as a followup series?

> +int
> +xchk_setup_fscounters(
> +	struct xfs_scrub	*sc,
> +	struct xfs_inode	*ip)
> +{
> +	int			error;
> +
> +	sc->buf = kmem_zalloc(sizeof(struct xchk_fscounters), KM_SLEEP);
> +	if (!sc->buf)
> +		return -ENOMEM;
> +
> +	/* We must get the incore counters set up before we can proceed. */
> +	error = xchk_fsc_warmup(sc);
> +	if (error)
> +		return error;
> +
> +	/*
> +	 * Pause background reclaim while we're scrubbing to reduce the
> +	 * likelihood of background perturbations to the counters throwing
> +	 * off our calculations.
> +	 */
> +	xchk_stop_reaping(sc);
> +
> +	return xchk_trans_alloc(sc, 0);
> +}
> +
> +/*
> + * Calculate what the global in-core counters ought to be from the incore
> + * per-AG structure.  Callers can compare this to the actual in-core counters
> + * to estimate by how much both in-core and on-disk counters need to be
> + * adjusted.
> + */
> +STATIC int
> +xchk_fsc_calc(
> +	struct xfs_scrub	*sc,
> +	struct xchk_fscounters	*fsc)
> +{
> +	struct xfs_mount	*mp = sc->mp;
> +	struct xfs_perag	*pag;
> +	unsigned long long	min_icount, max_icount;
> +	uint64_t		delayed;
> +	xfs_agnumber_t		agno;
> +	int			tries = 8;
> +
> +retry:
> +	fsc->icount = 0;
> +	fsc->ifree = 0;
> +	fsc->fdblocks = 0;
> +
> +	for (agno = 0; agno < mp->m_sb.sb_agcount; agno++) {
> +		pag = xfs_perag_get(mp, agno);
> +
> +		/* This somehow got unset since the warmup? */
> +		if (!pag->pagi_init || !pag->pagf_init) {
> +			xfs_perag_put(pag);
> +			return -EFSCORRUPTED;
> +		}
> +
> +		/* Count all the inodes */
> +		fsc->icount += pag->pagi_count;
> +		fsc->ifree += pag->pagi_freecount;
> +
> +		/* Add up the free/freelist/bnobt/cntbt blocks */
> +		fsc->fdblocks += pag->pagf_freeblks;
> +		fsc->fdblocks += pag->pagf_flcount;
> +		fsc->fdblocks += pag->pagf_btreeblks;
> +
> +		/*
> +		 * Per-AG reservations are taken out of the incore counters,
> +		 * so they must be left out of the free blocks computation.
> +		 */
> +		fsc->fdblocks -= pag->pag_meta_resv.ar_reserved;
> +		fsc->fdblocks -= pag->pag_rmapbt_resv.ar_orig_reserved;
> +
> +		xfs_perag_put(pag);
> +	}
> +
> +	/*
> +	 * The global incore space reservation is taken from the incore
> +	 * counters, so leave that out of the computation.
> +	 */
> +	fsc->fdblocks -= mp->m_resblks_avail;
> +
> +	/*
> +	 * Delayed allocation reservations are taken out of the incore counters
> +	 * but not recorded on disk, so leave them and their indlen blocks out
> +	 * of the computation.
> +	 */
> +	delayed = percpu_counter_sum(&mp->m_delalloc_blks);
> +	fsc->fdblocks -= delayed;
> +
> +	trace_xchk_fscounters_calc(mp, fsc->icount, fsc->ifree, fsc->fdblocks,
> +			delayed);
> +
> +	xfs_icount_range(mp, &min_icount, &max_icount);
> +
> +	/* Bail out if the values we compute are totally nonsense. */
> +	if (!xfs_verify_icount(mp, fsc->icount) ||
> +	    fsc->fdblocks > mp->m_sb.sb_dblocks ||
> +	    fsc->ifree > max_icount)

This is basically running xfs_icount_range() twice. Get rid of the 
xfs_verify_icount() helper and just open code it in the two places
that use it?

Oh, actually, the other place that uses it (xfs_validate_sb_write)
checks ifree against sb->sb_icount. I think that for this summation,
fsc->ifree shoudl never be greater than fsc->icount, right? Because
for every AG that is aggregated, pagi_freecount should always be
less than or equal to pagi_count, yes?

Hence I think the call to xfs_icount_range() can go away here, and
we just use fsc->ifree > fsc->icount...

> +		return -EFSCORRUPTED;
> +
> +	/*
> +	 * If ifree > icount then we probably had some perturbation in the
> +	 * counters while we were calculating things.  We'll try a few times
> +	 * to maintain ifree <= icount before giving up.
> +	 */
> +	if (fsc->ifree > fsc->icount) {
> +		if (tries--)
> +			goto retry;
> +		return -EFSCORRUPTED;

Oh, which you do here, anyway. When do you see perturbations that
need retries like this? Seems extremely unlikely...

> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * Is the @counter reasonably close to the @expected value?
> + *
> + * We neither locked nor froze anything in the filesystem while calculating
> + * @expected, which means that the counter could have changed.  We know the
> + * @old_value of the counter, which means that we also know how much the counter
> + * changed while calculating the expected value, and we'll take that as a rough
> + * guess at how accurate we have to be.  The expected value calculation runs in
> + * memory and never does any IO, so the window ought to be pretty small.
> + *
> + * First, ensure that @counter must never be negative.
> + *
> + * Next, compute the variance from the expected value that we'll accept.
> + * For now we'll use twice the change in the counter from start to finish
> + * or the minimum variance, whichever is larger.
> + */
> +static inline bool
> +xchk_fsc_within_range(
> +	struct xfs_scrub	*sc,
> +	const int64_t		old_value,
> +	struct percpu_counter	*counter,
> +	uint64_t		expected)
> +{
> +	int64_t			curr_value = percpu_counter_sum(counter);
> +	int64_t			range;
> +
> +	range = abs(2 * (curr_value - old_value));
> +	if (range < XCHK_FSC_MIN_VARIANCE)
> +		range = XCHK_FSC_MIN_VARIANCE;

Hmmmm. this is the difference between two summations a very short
period apart, but that doesn't really mean anything w.r.t. the
@expected value which was calculated over a much longer period of
time. e.g. aggregating over 10,000 AGs vs summing over 4 CPUs and
after 5,000 AGs were aggregated a huge file across AGs 10-1000 was
removed. The variance of the per-cpu counters is going to be
miniscule compared to the difference between the AG summation and
the per-cpu counter, but the fact is that both are correct and there
is no corruption to report.

I suspect what we should be doing is taking a summation before
we do the per-ag aggregation, followed by a summation after the per-ag
agg. If the per-ag result doesn't land inside the the pre- and post-
per-cpu counter summations, then we should only report a problem if
if the two per-cpu summations show no significant difference in values.

i.e. only if the pre- and post- summations are somewhat near to each
other in value should we consider the per-AG aggregation value to be
out of range, otherwise we should run the aggregation again. If the
per-cpu summations continue to have too much change to validate the
per-ag aggregation, then we should simply return "could not be
validated" rather than "something is wrong"....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com