[PATCH v2 00/36] xfsprogs-5.0: fix various problems

[PATCH v2 00/36] xfsprogs-5.0: fix various problems

[Darrick J. Wong]

Hi all,

Here are some fixes for xfsprogs 5.0:

Patch 1 fixes a deadlock in xfs_repair when we fail to flush an inode.
It's already on the list, but I pulled it in as a place to branch off
the upstream repo.

Patches 2-6 fix some errors in the autoconf scripts and tidy up some
of the debian packaging scripts.

Patches 7-9 are bugfixes and minor enhancements for xfs_io commands.

Patch 10 fixes xfs_scrub_all to discover the fs to physical device
mappings correctly when nvme devices are present in the system.

Patch 11 holds off automated xfs_scrub_all triggers until the system is
fully booted so that we don't end up racing metadata scrubbing with
system boot.

Patches 12-13 fix thread count discovery in xfs_scrub so that we stop
shadowing variables and estimate parallelism based at least somewhat on
the storage hardware and not just the number of CPUs.  In other words,
we'll stop pounding a spinning disk with all the CPUs.

Patches 14-15 restructure the read-verify pools to make them per-device
so that a media scan on heterogeneous storage won't flood a device with
low IOPS capacity.

Patch 16 teaches scrub not to complain if it can't close the mountpoint
after failing to open it.

Patch 17 teaches scrub to check filesystem labels for misleading
character sequences.

Patch 18 fixes the problem of xfs_scrub's support scripts not getting
rebuilt when file path definitions change.

Patch 19 teaches mkfs to validate extent size hint parameters so that
we cannot format filesystems that immediately fail to mount.

Patches 20-21 fix some finobt problems in xfs_db.

Patch 22 teaches xfs_info to be smarter about figuring out whether it
should be using a live query via xfs_spaceman or xfs_db.

Patches 23-24 fix some link count handling in xfs_repair when we're
(re)initializing the root directory and lost+found directories.

Patch 25 fixes some build warnings in xfs_repair.

Patch 26-27 refactor parts of xfs_repair to make it easier to pass
buffer ops pointers through the scan and rebuild functions.  These
cleanups should make it easier to adapt to the separation of buf_ops
for bnobt/cntbt and inobt/finobt that will appear in 5.1.

Patch 28-29 fix a crash in xfs_repair where accidentally create
duplicate rmapbt records for blocks that are initially allocated to the
free space btrees but then are freed back to the AGFL. while rebuilding
the rmap btree.

Patch 30 fixes a problem where xfs_repair phase 6 will fail to flush
partially corrected inode core data to the inode cluster buffer because
phase 6 hasn't yet fixed every thing that's wrong with the inode.  This
patch is part of a continuing series of fixes for Arkadiusz Miśkiewicz's
broken filesystem.

Patches 31-36 refactor the userspace buffer and inode log item handling
so that the item lifetimes make more sense -- they don't need to be kept
around after a transaction commits (or cancels), and we certainly don't
need to be leaking them because we got lazy about cleaning them up.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=xfsprogs-5.0-fixes

[PATCH 01/36] libxfs: fix repair deadlock due to failed inode flushes.

[Darrick J. Wong]

From: Dave Chinner <dchinner@redhat.com>

If inode_item_done() fails to flush an inode after we've grabbed a
reference to the underlying buffer during a transaction commit, we
fail to put the buffer and hence leak it. We then deadlock on the
next lookup ofthe inode buffer as it is still locked and no-one owns
it.

To fix it, put the buffer on error so that it gets unlocked and
can be recovered appropriately in a later phase of repair.

Reported-by: Arkadiusz Miskiewicz <arekm@maven.pl>
Fixes: d15188a1ec14 ("xfs: rework the inline directory verifiers")
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 libxfs/trans.c |   17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)


diff --git a/libxfs/trans.c b/libxfs/trans.c
index 46ff8b4a..10a35dd4 100644
--- a/libxfs/trans.c
+++ b/libxfs/trans.c
@@ -824,8 +824,10 @@ _("Transaction block reservation exceeded! %u > %u\n"),
 
 /*
  * Transaction commital code follows (i.e. write to disk in libxfs)
+ *
+ * XXX (dgc): should failure to flush the inode (e.g. due to uncorrected
+ * corruption) result in transaction commit failure w/ EFSCORRUPTED?
  */
-
 static void
 inode_item_done(
 	xfs_inode_log_item_t	*iip)
@@ -856,17 +858,24 @@ inode_item_done(
 		return;
 	}
 
+	/*
+	 * Flush the inode and disassociate it from the transaction regardless
+	 * of whether the flush succeed or not. If we fail the flush, make sure
+	 * we still release the buffer reference we currently hold.
+	 */
 	bp->b_log_item = iip;
 	error = libxfs_iflush_int(ip, bp);
+	ip->i_transp = NULL;	/* disassociate from transaction */
+	bp->b_log_item = NULL;	/* remove log item */
+	bp->b_transp = NULL;	/* remove xact ptr */
+
 	if (error) {
 		fprintf(stderr, _("%s: warning - iflush_int failed (%d)\n"),
 			progname, error);
+		libxfs_putbuf(bp);
 		return;
 	}
 
-	ip->i_transp = NULL;	/* disassociate from transaction */
-	bp->b_log_item = NULL;	/* remove log item */
-	bp->b_transp = NULL;	/* remove xact ptr */
 	libxfs_writebuf(bp, 0);
 #ifdef XACT_DEBUG
 	fprintf(stderr, "flushing dirty inode %llu, buffer %p\n",

Re: [PATCH 01/36] libxfs: fix repair deadlock due to failed inode flushes.

[Allison Henderson]

Looks good.

Reviewed-by: Allison Henderson <allison.henderson@oracle.com>

On 3/14/19 2:03 PM, Darrick J. Wong wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> If inode_item_done() fails to flush an inode after we've grabbed a
> reference to the underlying buffer during a transaction commit, we
> fail to put the buffer and hence leak it. We then deadlock on the
> next lookup ofthe inode buffer as it is still locked and no-one owns
> it.
> 
> To fix it, put the buffer on error so that it gets unlocked and
> can be recovered appropriately in a later phase of repair.
> 
> Reported-by: Arkadiusz Miskiewicz <arekm@maven.pl>
> Fixes: d15188a1ec14 ("xfs: rework the inline directory verifiers")
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   libxfs/trans.c |   17 +++++++++++++----
>   1 file changed, 13 insertions(+), 4 deletions(-)
> 
> 
> diff --git a/libxfs/trans.c b/libxfs/trans.c
> index 46ff8b4a..10a35dd4 100644
> --- a/libxfs/trans.c
> +++ b/libxfs/trans.c
> @@ -824,8 +824,10 @@ _("Transaction block reservation exceeded! %u > %u\n"),
>   
>   /*
>    * Transaction commital code follows (i.e. write to disk in libxfs)
> + *
> + * XXX (dgc): should failure to flush the inode (e.g. due to uncorrected
> + * corruption) result in transaction commit failure w/ EFSCORRUPTED?
>    */
> -
>   static void
>   inode_item_done(
>   	xfs_inode_log_item_t	*iip)
> @@ -856,17 +858,24 @@ inode_item_done(
>   		return;
>   	}
>   
> +	/*
> +	 * Flush the inode and disassociate it from the transaction regardless
> +	 * of whether the flush succeed or not. If we fail the flush, make sure
> +	 * we still release the buffer reference we currently hold.
> +	 */
>   	bp->b_log_item = iip;
>   	error = libxfs_iflush_int(ip, bp);
> +	ip->i_transp = NULL;	/* disassociate from transaction */
> +	bp->b_log_item = NULL;	/* remove log item */
> +	bp->b_transp = NULL;	/* remove xact ptr */
> +
>   	if (error) {
>   		fprintf(stderr, _("%s: warning - iflush_int failed (%d)\n"),
>   			progname, error);
> +		libxfs_putbuf(bp);
>   		return;
>   	}
>   
> -	ip->i_transp = NULL;	/* disassociate from transaction */
> -	bp->b_log_item = NULL;	/* remove log item */
> -	bp->b_transp = NULL;	/* remove xact ptr */
>   	libxfs_writebuf(bp, 0);
>   #ifdef XACT_DEBUG
>   	fprintf(stderr, "flushing dirty inode %llu, buffer %p\n",
> 

[PATCH 02/36] configure: use sys/xattr.h for fsetxattr detection

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

The only user of fsetxattr and HAVE_FSETXATTR is fsr, which includes
sys/xattr.h (from libc).  However, the m4 macro to detect fsetxattr
support requires attr/xattr.h (from libattr).  libattr dropped xattr.h
last year, so update the check.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 m4/package_libcdev.m4 |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4
index 7ee4acdd..2c0c72ce 100644
--- a/m4/package_libcdev.m4
+++ b/m4/package_libcdev.m4
@@ -223,14 +223,14 @@ AC_DEFUN([AC_HAVE_FLS],
   ])
 
 #
-# Check if we have a fsetxattr call (Mac OS X)
+# Check if we have a fsetxattr call
 #
 AC_DEFUN([AC_HAVE_FSETXATTR],
   [ AC_CHECK_DECL([fsetxattr],
        have_fsetxattr=yes,
        [],
        [#include <sys/types.h>
-        #include <attr/xattr.h>]
+        #include <sys/xattr.h>]
        )
     AC_SUBST(have_fsetxattr)
   ])

Re: [PATCH 02/36] configure: use sys/xattr.h for fsetxattr detection

[Allison Henderson]

Looks ok to me.

Reviewed-by: Allison Henderson <allison.henderson@oracle.com>

On 3/14/19 2:03 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> The only user of fsetxattr and HAVE_FSETXATTR is fsr, which includes
> sys/xattr.h (from libc).  However, the m4 macro to detect fsetxattr
> support requires attr/xattr.h (from libattr).  libattr dropped xattr.h
> last year, so update the check.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> ---
>   m4/package_libcdev.m4 |    4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4
> index 7ee4acdd..2c0c72ce 100644
> --- a/m4/package_libcdev.m4
> +++ b/m4/package_libcdev.m4
> @@ -223,14 +223,14 @@ AC_DEFUN([AC_HAVE_FLS],
>     ])
>   
>   #
> -# Check if we have a fsetxattr call (Mac OS X)
> +# Check if we have a fsetxattr call
>   #
>   AC_DEFUN([AC_HAVE_FSETXATTR],
>     [ AC_CHECK_DECL([fsetxattr],
>          have_fsetxattr=yes,
>          [],
>          [#include <sys/types.h>
> -        #include <attr/xattr.h>]
> +        #include <sys/xattr.h>]
>          )
>       AC_SUBST(have_fsetxattr)
>     ])
> 

[PATCH 03/36] configure.ac: fix alignment of features

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix the alignment of the feature options in the --help screen.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 configure.ac |   28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)


diff --git a/configure.ac b/configure.ac
index ccc6e292..5a4e55ad 100644
--- a/configure.ac
+++ b/configure.ac
@@ -28,82 +28,82 @@ if test "${BUILD_CFLAGS+set}" != "set"; then
 fi
 
 AC_ARG_ENABLE(shared,
-[ --enable-shared=[yes/no] Enable use of shared libraries [default=yes]],,
+[  --enable-shared=[yes/no]  Enable use of shared libraries [default=yes]],,
 	enable_shared=yes)
 AC_SUBST(enable_shared)
 
 AC_ARG_ENABLE(gettext,
-[ --enable-gettext=[yes/no] Enable alternate language support [default=yes]],,
+[  --enable-gettext=[yes/no] Enable alternate language support [default=yes]],,
 	enable_gettext=yes)
 AC_SUBST(enable_gettext)
 
 AC_ARG_ENABLE(blkid,
-[ --enable-blkid=[yes/no] Enable use of block device id library [default=yes]],,
+[  --enable-blkid=[yes/no]   Enable use of block device id library [default=yes]],,
 	enable_blkid=yes)
 AC_SUBST(enable_blkid)
 
 AC_ARG_ENABLE(readline,
-[ --enable-readline=[yes/no] Enable readline command editing [default=no]],
+[  --enable-readline=[yes/no] Enable readline command editing [default=no]],
 	test $enable_readline = yes && libreadline="-lreadline",
 	enable_readline=no)
 AC_SUBST(libreadline)
 AC_SUBST(enable_readline)
 
 AC_ARG_ENABLE(editline,
-[ --enable-editline=[yes/no] Enable editline command editing [default=no]],
+[  --enable-editline=[yes/no] Enable editline command editing [default=no]],
 	test $enable_editline = yes && libeditline="-ledit",
 	enable_editline=no)
 AC_SUBST(libeditline)
 AC_SUBST(enable_editline)
 
 AC_ARG_ENABLE(termcap,
-[ --enable-termcap=[yes/no] Enable terminal capabilities library [default=no]],
+[  --enable-termcap=[yes/no] Enable terminal capabilities library [default=no]],
 	test $enable_termcap = yes && libtermcap="-ltermcap",)
 AC_SUBST(libtermcap)
 
 
 AC_ARG_ENABLE(lib64,
-[ --enable-lib64=[yes/no] Enable lib64 support [default=yes]],,
+[  --enable-lib64=[yes/no]   Enable lib64 support [default=yes]],,
 	enable_lib64=yes)
 AC_SUBST(enable_lib64)
 
 AC_ARG_ENABLE(librt,
-[ --enable-librt=[yes/no] Enable librt support [default=yes]],,
+[  --enable-librt=[yes/no]   Enable librt support [default=yes]],,
 	enable_librt=yes)
 AC_SUBST(enable_librt)
 
 # Enable UBSAN; set enable_ubsan=probe below to enable autoprobe.
 AC_ARG_ENABLE(ubsan,
-[ --enable-ubsan=[yes/no] Enable Undefined Behavior Sanitizer (UBSAN) [default=no]],,
+[  --enable-ubsan=[yes/no]   Enable Undefined Behavior Sanitizer (UBSAN) [default=no]],,
 	enable_ubsan=no)
 AC_SUBST(enable_ubsan)
 
 # Enable ADDRSAN; set enable_addrsan=probe below to enable autoprobe.
 AC_ARG_ENABLE(addrsan,
-[ --enable-addrsan=[yes/no] Enable Address Sanitizer (ADDRSAN) [default=no]],,
+[  --enable-addrsan=[yes/no] Enable Address Sanitizer (ADDRSAN) [default=no]],,
 	enable_addrsan=no)
 AC_SUBST(enable_addrsan)
 
 # Enable THREADSAN; set enable_threadsan=probe to enable autoprobe.
 AC_ARG_ENABLE(threadsan,
-[ --enable-threadsan=[yes/no] Enable Thread Sanitizer (THREADSAN) [default=no]],,
+[  --enable-threadsan=[yes/no] Enable Thread Sanitizer (THREADSAN) [default=no]],,
 	enable_threadsan=no)
 AC_SUBST(enable_threadsan)
 
 AC_ARG_ENABLE(lto,
-[ --enable-lto=[yes/no]      Enable link time optimization (LTO) [default=no]],,
+[  --enable-lto=[yes/no]     Enable link time optimization (LTO) [default=no]],,
 	enable_lto=no)
 AC_SUBST(enable_lto)
 
 # Enable xfs_scrub build
 AC_ARG_ENABLE(scrub,
-[ --enable-scrub=[yes/no]    Enable build of xfs_scrub utility [default=yes]],,
+[  --enable-scrub=[yes/no]   Enable build of xfs_scrub utility [default=yes]],,
 	enable_scrub=yes)
 AC_SUBST(enable_scrub)
 
 # Enable libicu for xfs_scrubbing of malicious unicode sequences in names
 AC_ARG_ENABLE(libicu,
-[ --enable-libicu=[yes/no]   Enable Unicode name scanning in xfs_scrub (libicu) [default=probe]],,
+[  --enable-libicu=[yes/no]  Enable Unicode name scanning in xfs_scrub (libicu) [default=probe]],,
 	enable_libicu=probe)
 
 #

Re: [PATCH 03/36] configure.ac: fix alignment of features

[Eric Sandeen]

On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fix the alignment of the feature options in the --help screen.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Beauty!

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  configure.ac |   28 ++++++++++++++--------------
>  1 file changed, 14 insertions(+), 14 deletions(-)
> 
> 
> diff --git a/configure.ac b/configure.ac
> index ccc6e292..5a4e55ad 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -28,82 +28,82 @@ if test "${BUILD_CFLAGS+set}" != "set"; then
>  fi
>  
>  AC_ARG_ENABLE(shared,
> -[ --enable-shared=[yes/no] Enable use of shared libraries [default=yes]],,
> +[  --enable-shared=[yes/no]  Enable use of shared libraries [default=yes]],,
>  	enable_shared=yes)
>  AC_SUBST(enable_shared)
>  
>  AC_ARG_ENABLE(gettext,
> -[ --enable-gettext=[yes/no] Enable alternate language support [default=yes]],,
> +[  --enable-gettext=[yes/no] Enable alternate language support [default=yes]],,
>  	enable_gettext=yes)
>  AC_SUBST(enable_gettext)
>  
>  AC_ARG_ENABLE(blkid,
> -[ --enable-blkid=[yes/no] Enable use of block device id library [default=yes]],,
> +[  --enable-blkid=[yes/no]   Enable use of block device id library [default=yes]],,
>  	enable_blkid=yes)
>  AC_SUBST(enable_blkid)
>  
>  AC_ARG_ENABLE(readline,
> -[ --enable-readline=[yes/no] Enable readline command editing [default=no]],
> +[  --enable-readline=[yes/no] Enable readline command editing [default=no]],
>  	test $enable_readline = yes && libreadline="-lreadline",
>  	enable_readline=no)
>  AC_SUBST(libreadline)
>  AC_SUBST(enable_readline)
>  
>  AC_ARG_ENABLE(editline,
> -[ --enable-editline=[yes/no] Enable editline command editing [default=no]],
> +[  --enable-editline=[yes/no] Enable editline command editing [default=no]],
>  	test $enable_editline = yes && libeditline="-ledit",
>  	enable_editline=no)
>  AC_SUBST(libeditline)
>  AC_SUBST(enable_editline)
>  
>  AC_ARG_ENABLE(termcap,
> -[ --enable-termcap=[yes/no] Enable terminal capabilities library [default=no]],
> +[  --enable-termcap=[yes/no] Enable terminal capabilities library [default=no]],
>  	test $enable_termcap = yes && libtermcap="-ltermcap",)
>  AC_SUBST(libtermcap)
>  
>  
>  AC_ARG_ENABLE(lib64,
> -[ --enable-lib64=[yes/no] Enable lib64 support [default=yes]],,
> +[  --enable-lib64=[yes/no]   Enable lib64 support [default=yes]],,
>  	enable_lib64=yes)
>  AC_SUBST(enable_lib64)
>  
>  AC_ARG_ENABLE(librt,
> -[ --enable-librt=[yes/no] Enable librt support [default=yes]],,
> +[  --enable-librt=[yes/no]   Enable librt support [default=yes]],,
>  	enable_librt=yes)
>  AC_SUBST(enable_librt)
>  
>  # Enable UBSAN; set enable_ubsan=probe below to enable autoprobe.
>  AC_ARG_ENABLE(ubsan,
> -[ --enable-ubsan=[yes/no] Enable Undefined Behavior Sanitizer (UBSAN) [default=no]],,
> +[  --enable-ubsan=[yes/no]   Enable Undefined Behavior Sanitizer (UBSAN) [default=no]],,
>  	enable_ubsan=no)
>  AC_SUBST(enable_ubsan)
>  
>  # Enable ADDRSAN; set enable_addrsan=probe below to enable autoprobe.
>  AC_ARG_ENABLE(addrsan,
> -[ --enable-addrsan=[yes/no] Enable Address Sanitizer (ADDRSAN) [default=no]],,
> +[  --enable-addrsan=[yes/no] Enable Address Sanitizer (ADDRSAN) [default=no]],,
>  	enable_addrsan=no)
>  AC_SUBST(enable_addrsan)
>  
>  # Enable THREADSAN; set enable_threadsan=probe to enable autoprobe.
>  AC_ARG_ENABLE(threadsan,
> -[ --enable-threadsan=[yes/no] Enable Thread Sanitizer (THREADSAN) [default=no]],,
> +[  --enable-threadsan=[yes/no] Enable Thread Sanitizer (THREADSAN) [default=no]],,
>  	enable_threadsan=no)
>  AC_SUBST(enable_threadsan)
>  
>  AC_ARG_ENABLE(lto,
> -[ --enable-lto=[yes/no]      Enable link time optimization (LTO) [default=no]],,
> +[  --enable-lto=[yes/no]     Enable link time optimization (LTO) [default=no]],,
>  	enable_lto=no)
>  AC_SUBST(enable_lto)
>  
>  # Enable xfs_scrub build
>  AC_ARG_ENABLE(scrub,
> -[ --enable-scrub=[yes/no]    Enable build of xfs_scrub utility [default=yes]],,
> +[  --enable-scrub=[yes/no]   Enable build of xfs_scrub utility [default=yes]],,
>  	enable_scrub=yes)
>  AC_SUBST(enable_scrub)
>  
>  # Enable libicu for xfs_scrubbing of malicious unicode sequences in names
>  AC_ARG_ENABLE(libicu,
> -[ --enable-libicu=[yes/no]   Enable Unicode name scanning in xfs_scrub (libicu) [default=probe]],,
> +[  --enable-libicu=[yes/no]  Enable Unicode name scanning in xfs_scrub (libicu) [default=probe]],,
>  	enable_libicu=probe)
>  
>  #
> 

Re: [PATCH 03/36] configure.ac: fix alignment of features

[Allison Henderson]

Looks good.  Thanks!

Reviewed-by: Allison Henderson <allison.henderson@oracle.com>

On 3/14/19 2:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fix the alignment of the feature options in the --help screen.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   configure.ac |   28 ++++++++++++++--------------
>   1 file changed, 14 insertions(+), 14 deletions(-)
> 
> 
> diff --git a/configure.ac b/configure.ac
> index ccc6e292..5a4e55ad 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -28,82 +28,82 @@ if test "${BUILD_CFLAGS+set}" != "set"; then
>   fi
>   
>   AC_ARG_ENABLE(shared,
> -[ --enable-shared=[yes/no] Enable use of shared libraries [default=yes]],,
> +[  --enable-shared=[yes/no]  Enable use of shared libraries [default=yes]],,
>   	enable_shared=yes)
>   AC_SUBST(enable_shared)
>   
>   AC_ARG_ENABLE(gettext,
> -[ --enable-gettext=[yes/no] Enable alternate language support [default=yes]],,
> +[  --enable-gettext=[yes/no] Enable alternate language support [default=yes]],,
>   	enable_gettext=yes)
>   AC_SUBST(enable_gettext)
>   
>   AC_ARG_ENABLE(blkid,
> -[ --enable-blkid=[yes/no] Enable use of block device id library [default=yes]],,
> +[  --enable-blkid=[yes/no]   Enable use of block device id library [default=yes]],,
>   	enable_blkid=yes)
>   AC_SUBST(enable_blkid)
>   
>   AC_ARG_ENABLE(readline,
> -[ --enable-readline=[yes/no] Enable readline command editing [default=no]],
> +[  --enable-readline=[yes/no] Enable readline command editing [default=no]],
>   	test $enable_readline = yes && libreadline="-lreadline",
>   	enable_readline=no)
>   AC_SUBST(libreadline)
>   AC_SUBST(enable_readline)
>   
>   AC_ARG_ENABLE(editline,
> -[ --enable-editline=[yes/no] Enable editline command editing [default=no]],
> +[  --enable-editline=[yes/no] Enable editline command editing [default=no]],
>   	test $enable_editline = yes && libeditline="-ledit",
>   	enable_editline=no)
>   AC_SUBST(libeditline)
>   AC_SUBST(enable_editline)
>   
>   AC_ARG_ENABLE(termcap,
> -[ --enable-termcap=[yes/no] Enable terminal capabilities library [default=no]],
> +[  --enable-termcap=[yes/no] Enable terminal capabilities library [default=no]],
>   	test $enable_termcap = yes && libtermcap="-ltermcap",)
>   AC_SUBST(libtermcap)
>   
>   
>   AC_ARG_ENABLE(lib64,
> -[ --enable-lib64=[yes/no] Enable lib64 support [default=yes]],,
> +[  --enable-lib64=[yes/no]   Enable lib64 support [default=yes]],,
>   	enable_lib64=yes)
>   AC_SUBST(enable_lib64)
>   
>   AC_ARG_ENABLE(librt,
> -[ --enable-librt=[yes/no] Enable librt support [default=yes]],,
> +[  --enable-librt=[yes/no]   Enable librt support [default=yes]],,
>   	enable_librt=yes)
>   AC_SUBST(enable_librt)
>   
>   # Enable UBSAN; set enable_ubsan=probe below to enable autoprobe.
>   AC_ARG_ENABLE(ubsan,
> -[ --enable-ubsan=[yes/no] Enable Undefined Behavior Sanitizer (UBSAN) [default=no]],,
> +[  --enable-ubsan=[yes/no]   Enable Undefined Behavior Sanitizer (UBSAN) [default=no]],,
>   	enable_ubsan=no)
>   AC_SUBST(enable_ubsan)
>   
>   # Enable ADDRSAN; set enable_addrsan=probe below to enable autoprobe.
>   AC_ARG_ENABLE(addrsan,
> -[ --enable-addrsan=[yes/no] Enable Address Sanitizer (ADDRSAN) [default=no]],,
> +[  --enable-addrsan=[yes/no] Enable Address Sanitizer (ADDRSAN) [default=no]],,
>   	enable_addrsan=no)
>   AC_SUBST(enable_addrsan)
>   
>   # Enable THREADSAN; set enable_threadsan=probe to enable autoprobe.
>   AC_ARG_ENABLE(threadsan,
> -[ --enable-threadsan=[yes/no] Enable Thread Sanitizer (THREADSAN) [default=no]],,
> +[  --enable-threadsan=[yes/no] Enable Thread Sanitizer (THREADSAN) [default=no]],,
>   	enable_threadsan=no)
>   AC_SUBST(enable_threadsan)
>   
>   AC_ARG_ENABLE(lto,
> -[ --enable-lto=[yes/no]      Enable link time optimization (LTO) [default=no]],,
> +[  --enable-lto=[yes/no]     Enable link time optimization (LTO) [default=no]],,
>   	enable_lto=no)
>   AC_SUBST(enable_lto)
>   
>   # Enable xfs_scrub build
>   AC_ARG_ENABLE(scrub,
> -[ --enable-scrub=[yes/no]    Enable build of xfs_scrub utility [default=yes]],,
> +[  --enable-scrub=[yes/no]   Enable build of xfs_scrub utility [default=yes]],,
>   	enable_scrub=yes)
>   AC_SUBST(enable_scrub)
>   
>   # Enable libicu for xfs_scrubbing of malicious unicode sequences in names
>   AC_ARG_ENABLE(libicu,
> -[ --enable-libicu=[yes/no]   Enable Unicode name scanning in xfs_scrub (libicu) [default=probe]],,
> +[  --enable-libicu=[yes/no]  Enable Unicode name scanning in xfs_scrub (libicu) [default=probe]],,
>   	enable_libicu=probe)
>   
>   #
> 

[PATCH 04/36] debian: drop dangling libhandle.a symlinks in xfslibs-dev

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

We don't ship static libhandle libraries anymore, so make sure we drop
the symlink.

Fixes: ec1cf08dbeb2d ("Several updates to use more modern Debian packaging")
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debian/rules |    1 +
 1 file changed, 1 insertion(+)


diff --git a/debian/rules b/debian/rules
index 4c50654c..80dff4bb 100755
--- a/debian/rules
+++ b/debian/rules
@@ -80,6 +80,7 @@ binary-arch: checkroot built
 	install -D -m 0755 debian/local/initramfs.hook debian/xfsprogs/usr/share/initramfs-tools/hooks/xfs
 	rmdir debian/xfslibs-dev/usr/share/doc/xfsprogs
 	rm -f debian/xfslibs-dev/lib/libhandle.la
+	rm -f debian/xfslibs-dev/lib/libhandle.a
 	rm -fr debian/xfslibs-dev/usr/lib
 	dh_installdocs
 	dh_installchangelogs

Re: [PATCH 04/36] debian: drop dangling libhandle.a symlinks in xfslibs-dev

[Nathan Scott]

On Fri, Mar 15, 2019 at 8:04 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> We don't ship static libhandle libraries anymore, so make sure we drop
> the symlink.
>
> Fixes: ec1cf08dbeb2d ("Several updates to use more modern Debian packaging")
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

LGTM.

Reviewed-by: Nathan Scott <nathans@debian.org>

cheers.

--
Nathan

Re: [PATCH 04/36] debian: drop dangling libhandle.a symlinks in xfslibs-dev

[Allison Henderson]

Alighty, looks ok.

Reviewed-by: Allison Henderson <allison.henderson@oracle.com>

On 3/14/19 2:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> We don't ship static libhandle libraries anymore, so make sure we drop
> the symlink.
> 
> Fixes: ec1cf08dbeb2d ("Several updates to use more modern Debian packaging")
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   debian/rules |    1 +
>   1 file changed, 1 insertion(+)
> 
> 
> diff --git a/debian/rules b/debian/rules
> index 4c50654c..80dff4bb 100755
> --- a/debian/rules
> +++ b/debian/rules
> @@ -80,6 +80,7 @@ binary-arch: checkroot built
>   	install -D -m 0755 debian/local/initramfs.hook debian/xfsprogs/usr/share/initramfs-tools/hooks/xfs
>   	rmdir debian/xfslibs-dev/usr/share/doc/xfsprogs
>   	rm -f debian/xfslibs-dev/lib/libhandle.la
> +	rm -f debian/xfslibs-dev/lib/libhandle.a
>   	rm -fr debian/xfslibs-dev/usr/lib
>   	dh_installdocs
>   	dh_installchangelogs
> 

[PATCH 05/36] debian: don't bypass top level Makefile when building subdirs

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

The top level Makefile does some processing to set build environment
variables (Q and CHECK_CMD).  debian/rules uses -C to build subdirs
directly, which bypases this feature of the top-level makefile, which
causes more build spew than necessary (because Q never gets set to quiet
the build).

Since the top level makefile can be used to build the subdirs
debian/rules cares about, drop the -C and build subdirs via the top
level Makefile to quiet the build.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debian/rules |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/debian/rules b/debian/rules
index 80dff4bb..7c2a8b3c 100755
--- a/debian/rules
+++ b/debian/rules
@@ -51,7 +51,7 @@ dibuild:
 			$(MAKE) -C $$dir NODEP=1 install-headers; \
 		done; \
 		for dir in include libxfs libxcmd libfrog mkfs; do \
-			$(MAKE) -C $$dir; \
+			$(MAKE) $$dir; \
 		done; \
 		mv mkfs/mkfs.xfs mkfs/mkfs.xfs-$(bootpkg); \
 		$(MAKE) distclean; \

Re: [PATCH 05/36] debian: don't bypass top level Makefile when building subdirs

[Nathan Scott]

On Fri, Mar 15, 2019 at 8:04 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> The top level Makefile does some processing to set build environment
> variables (Q and CHECK_CMD).  debian/rules uses -C to build subdirs
> directly, which bypases this feature of the top-level makefile, which
> causes more build spew than necessary (because Q never gets set to quiet
> the build).
>
> Since the top level makefile can be used to build the subdirs
> debian/rules cares about, drop the -C and build subdirs via the top
> level Makefile to quiet the build.
>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Neat.

Reviewed-by: Nathan Scott <nathans@debian.org>

cheers.

--
Nathan

Re: [PATCH 05/36] debian: don't bypass top level Makefile when building subdirs

[Allison Henderson]

Looks ok.  Thanks!

Reviewed-by: Allison Henderson <allison.henderson@oracle.com>

On 3/14/19 2:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> The top level Makefile does some processing to set build environment
> variables (Q and CHECK_CMD).  debian/rules uses -C to build subdirs
> directly, which bypases this feature of the top-level makefile, which
> causes more build spew than necessary (because Q never gets set to quiet
> the build).
> 
> Since the top level makefile can be used to build the subdirs
> debian/rules cares about, drop the -C and build subdirs via the top
> level Makefile to quiet the build.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   debian/rules |    2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> 
> diff --git a/debian/rules b/debian/rules
> index 80dff4bb..7c2a8b3c 100755
> --- a/debian/rules
> +++ b/debian/rules
> @@ -51,7 +51,7 @@ dibuild:
>   			$(MAKE) -C $$dir NODEP=1 install-headers; \
>   		done; \
>   		for dir in include libxfs libxcmd libfrog mkfs; do \
> -			$(MAKE) -C $$dir; \
> +			$(MAKE) $$dir; \
>   		done; \
>   		mv mkfs/mkfs.xfs mkfs/mkfs.xfs-$(bootpkg); \
>   		$(MAKE) distclean; \
> 

[PATCH 06/36] debian: enable parallel make

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Use parallel make to speed up dpkg builds.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debian/rules |   13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)


diff --git a/debian/rules b/debian/rules
index 7c2a8b3c..e8509fb3 100755
--- a/debian/rules
+++ b/debian/rules
@@ -2,6 +2,11 @@
 
 export DH_VERBOSE=1
 
+ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
+    NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
+    PMAKEFLAGS += -j$(NUMJOBS)
+endif
+
 package = xfsprogs
 develop = xfslibs-dev
 bootpkg = xfsprogs-udeb
@@ -30,7 +35,7 @@ build-arch: built
 build-indep: built
 built: dibuild config
 	@echo "== dpkg-buildpackage: build" 1>&2
-	$(MAKE) default
+	$(MAKE) $(PMAKEFLAGS) default
 	touch built
 
 config: .census
@@ -38,7 +43,7 @@ config: .census
 	@echo "== dpkg-buildpackage: configure" 1>&2
 	$(checkdir)
 	AUTOHEADER=/bin/true dh_autoreconf
-	$(options) $(MAKE) include/platform_defs.h
+	$(options) $(MAKE) $(PMAKEFLAGS) include/platform_defs.h
 	touch .census
 
 dibuild:
@@ -48,10 +53,10 @@ dibuild:
 		$(diopts) $(MAKE) include/platform_defs.h; \
 		mkdir -p include/xfs; \
 		for dir in include libxfs; do \
-			$(MAKE) -C $$dir NODEP=1 install-headers; \
+			$(MAKE) $(PMAKEFLAGS) -C $$dir NODEP=1 install-headers; \
 		done; \
 		for dir in include libxfs libxcmd libfrog mkfs; do \
-			$(MAKE) $$dir; \
+			$(MAKE) $(PMAKEFLAGS) $$dir; \
 		done; \
 		mv mkfs/mkfs.xfs mkfs/mkfs.xfs-$(bootpkg); \
 		$(MAKE) distclean; \

Re: [PATCH 06/36] debian: enable parallel make

[Nathan Scott]

On Fri, Mar 15, 2019 at 8:04 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> Use parallel make to speed up dpkg builds.

Seems a little more involved than I woulda expected, but LGTM.

Reviewed-by: Nathan Scott <nathans@debian.org>

>                 for dir in include libxfs; do \
> -                       $(MAKE) -C $$dir NODEP=1 install-headers; \
> +                       $(MAKE) $(PMAKEFLAGS) -C $$dir NODEP=1 install-headers; \
>                 done; \

OOC, does the $(MAKE) -C above warrant the same treatment as the earlier patch?
(i.e. removing the -C there for less verbosity)

cheers.

--
Nathan

Re: [PATCH 06/36] debian: enable parallel make

[Darrick J. Wong]

On Fri, Mar 15, 2019 at 11:18:32AM +1100, Nathan Scott wrote:
> On Fri, Mar 15, 2019 at 8:04 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
> >
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> >
> > Use parallel make to speed up dpkg builds.
> 
> Seems a little more involved than I woulda expected, but LGTM.
> 
> Reviewed-by: Nathan Scott <nathans@debian.org>
> 
> >                 for dir in include libxfs; do \
> > -                       $(MAKE) -C $$dir NODEP=1 install-headers; \
> > +                       $(MAKE) $(PMAKEFLAGS) -C $$dir NODEP=1 install-headers; \
> >                 done; \
> 
> OOC, does the $(MAKE) -C above warrant the same treatment as the earlier patch?
> (i.e. removing the -C there for less verbosity)

Hmm, that seems like it could be the case.  I'll have a look tomorrow.

--D

> cheers.
> 
> --
> Nathan

Re: [PATCH 06/36] debian: enable parallel make

[Darrick J. Wong]

On Thu, Mar 14, 2019 at 06:01:13PM -0700, Darrick J. Wong wrote:
> On Fri, Mar 15, 2019 at 11:18:32AM +1100, Nathan Scott wrote:
> > On Fri, Mar 15, 2019 at 8:04 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
> > >
> > > From: Darrick J. Wong <darrick.wong@oracle.com>
> > >
> > > Use parallel make to speed up dpkg builds.
> > 
> > Seems a little more involved than I woulda expected, but LGTM.
> > 
> > Reviewed-by: Nathan Scott <nathans@debian.org>
> > 
> > >                 for dir in include libxfs; do \
> > > -                       $(MAKE) -C $$dir NODEP=1 install-headers; \
> > > +                       $(MAKE) $(PMAKEFLAGS) -C $$dir NODEP=1 install-headers; \
> > >                 done; \
> > 
> > OOC, does the $(MAKE) -C above warrant the same treatment as the earlier patch?
> > (i.e. removing the -C there for less verbosity)
> 
> Hmm, that seems like it could be the case.  I'll have a look tomorrow.

It doesn't work, because buildrules sets up $(SUBDIRS) as (phony)
targets so that you can do:

$ make scrub

and have the top level Makefile set Q and then recurse into scrub/
instead of doing it yourself ("make -C scrub").  That's why the previous
-C removal patch works.

The hunk of Makefile you're talking about does:

$ make -C include install-headers

which is different because install-headers is a target that only exists
in {include,libxfs}/Makefile, not the top level build system.  So we
have to get make to chdir into include/ for the target to work at all.

--D

> --D
> 
> > cheers.
> > 
> > --
> > Nathan

[PATCH 07/36] xfs_io: actually check copy file range helper return values

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

We need to check the return value of copy_src_filesize and
copy_dst_truncate because either could return -1 due to fstat/ftruncate
failure.

Fixes: 628e112afdd98c5 ("xfs_io: implement 'copy_range' command")
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 io/copy_file_range.c |   17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)


diff --git a/io/copy_file_range.c b/io/copy_file_range.c
index 4e2969c9..d069e5bb 100644
--- a/io/copy_file_range.c
+++ b/io/copy_file_range.c
@@ -120,11 +120,24 @@ copy_range_f(int argc, char **argv)
 		return 0;
 
 	if (src == 0 && dst == 0 && len == 0) {
-		len = copy_src_filesize(fd);
-		copy_dst_truncate();
+		off64_t	sz;
+
+		sz = copy_src_filesize(fd);
+		if (sz < 0 || (unsigned long long)sz > SIZE_MAX) {
+			ret = 1;
+			goto out;
+		}
+		len = sz;
+
+		ret = copy_dst_truncate();
+		if (ret < 0) {
+			ret = 1;
+			goto out;
+		}
 	}
 
 	ret = copy_file_range_cmd(fd, &src, &dst, len);
+out:
 	close(fd);
 	return ret;
 }

Re: [PATCH 07/36] xfs_io: actually check copy file range helper return values

[Eric Sandeen]

On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> We need to check the return value of copy_src_filesize and
> copy_dst_truncate because either could return -1 due to fstat/ftruncate
> failure.
> 
> Fixes: 628e112afdd98c5 ("xfs_io: implement 'copy_range' command")
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> Reviewed-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
> Reviewed-by: Christoph Hellwig <hch@lst.de>

I see this has reviews already, but I'm not conviced 1 is the right
return on error.

i.e. the error condition just prior returns 0:

        fd = openfile(argv[optind], NULL, IO_READONLY, 0, NULL);
        if (fd < 0)
                return 0;

but OTOH if copy_file_range_cmd() fails we return nonzero.  Argh.

Is there a rhyme or reason here that I'm missing?  Is it broken now
so just do a coinflip on the new return for now?


> ---
>  io/copy_file_range.c |   17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/io/copy_file_range.c b/io/copy_file_range.c
> index 4e2969c9..d069e5bb 100644
> --- a/io/copy_file_range.c
> +++ b/io/copy_file_range.c
> @@ -120,11 +120,24 @@ copy_range_f(int argc, char **argv)
>  		return 0;
>  
>  	if (src == 0 && dst == 0 && len == 0) {
> -		len = copy_src_filesize(fd);
> -		copy_dst_truncate();
> +		off64_t	sz;
> +
> +		sz = copy_src_filesize(fd);
> +		if (sz < 0 || (unsigned long long)sz > SIZE_MAX) {
> +			ret = 1;
> +			goto out;
> +		}
> +		len = sz;
> +
> +		ret = copy_dst_truncate();
> +		if (ret < 0) {
> +			ret = 1;
> +			goto out;
> +		}
>  	}
>  
>  	ret = copy_file_range_cmd(fd, &src, &dst, len);
> +out:
>  	close(fd);
>  	return ret;
>  }
> 

Re: [PATCH 07/36] xfs_io: actually check copy file range helper return values

[Darrick J. Wong]

On Thu, Mar 14, 2019 at 09:12:11PM -0500, Eric Sandeen wrote:
> On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > We need to check the return value of copy_src_filesize and
> > copy_dst_truncate because either could return -1 due to fstat/ftruncate
> > failure.
> > 
> > Fixes: 628e112afdd98c5 ("xfs_io: implement 'copy_range' command")
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > Reviewed-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
> > Reviewed-by: Christoph Hellwig <hch@lst.de>
> 
> I see this has reviews already, but I'm not conviced 1 is the right
> return on error.
> 
> i.e. the error condition just prior returns 0:
> 
>         fd = openfile(argv[optind], NULL, IO_READONLY, 0, NULL);
>         if (fd < 0)
>                 return 0;
> 
> but OTOH if copy_file_range_cmd() fails we return nonzero.  Argh.
> 
> Is there a rhyme or reason here that I'm missing?  Is it broken now
> so just do a coinflip on the new return for now?

Pretty much. :(

I don't really care if you change the return value, I just thought it
was bad form not to check the syscall/helper return values at all.

--D

> 
> > ---
> >  io/copy_file_range.c |   17 +++++++++++++++--
> >  1 file changed, 15 insertions(+), 2 deletions(-)
> > 
> > 
> > diff --git a/io/copy_file_range.c b/io/copy_file_range.c
> > index 4e2969c9..d069e5bb 100644
> > --- a/io/copy_file_range.c
> > +++ b/io/copy_file_range.c
> > @@ -120,11 +120,24 @@ copy_range_f(int argc, char **argv)
> >  		return 0;
> >  
> >  	if (src == 0 && dst == 0 && len == 0) {
> > -		len = copy_src_filesize(fd);
> > -		copy_dst_truncate();
> > +		off64_t	sz;
> > +
> > +		sz = copy_src_filesize(fd);
> > +		if (sz < 0 || (unsigned long long)sz > SIZE_MAX) {
> > +			ret = 1;
> > +			goto out;
> > +		}
> > +		len = sz;
> > +
> > +		ret = copy_dst_truncate();
> > +		if (ret < 0) {
> > +			ret = 1;
> > +			goto out;
> > +		}
> >  	}
> >  
> >  	ret = copy_file_range_cmd(fd, &src, &dst, len);
> > +out:
> >  	close(fd);
> >  	return ret;
> >  }
> > 

Re: [PATCH 07/36] xfs_io: actually check copy file range helper return values

[Eric Sandeen]

On 3/14/19 9:56 PM, Darrick J. Wong wrote:
> On Thu, Mar 14, 2019 at 09:12:11PM -0500, Eric Sandeen wrote:
>> On 3/14/19 4:04 PM, Darrick J. Wong wrote:
>>> From: Darrick J. Wong <darrick.wong@oracle.com>
>>>
>>> We need to check the return value of copy_src_filesize and
>>> copy_dst_truncate because either could return -1 due to fstat/ftruncate
>>> failure.
>>>
>>> Fixes: 628e112afdd98c5 ("xfs_io: implement 'copy_range' command")
>>> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
>>> Reviewed-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
>>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>>
>> I see this has reviews already, but I'm not conviced 1 is the right
>> return on error.
>>
>> i.e. the error condition just prior returns 0:
>>
>>         fd = openfile(argv[optind], NULL, IO_READONLY, 0, NULL);
>>         if (fd < 0)
>>                 return 0;
>>
>> but OTOH if copy_file_range_cmd() fails we return nonzero.  Argh.
>>
>> Is there a rhyme or reason here that I'm missing?  Is it broken now
>> so just do a coinflip on the new return for now?
> 
> Pretty much. :(
> 
> I don't really care if you change the return value, I just thought it
> was bad form not to check the syscall/helper return values at all.

Yeah no argument there.  Ok, so this function already returns inconsistently,
I'll worry about that later (I hope)

Thanks,
-Eric

Re: [PATCH 07/36] xfs_io: actually check copy file range helper return values

[Dave Chinner]

On Fri, Mar 15, 2019 at 11:51:56AM -0500, Eric Sandeen wrote:
> 
> 
> On 3/14/19 9:56 PM, Darrick J. Wong wrote:
> > On Thu, Mar 14, 2019 at 09:12:11PM -0500, Eric Sandeen wrote:
> >> On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> >>> From: Darrick J. Wong <darrick.wong@oracle.com>
> >>>
> >>> We need to check the return value of copy_src_filesize and
> >>> copy_dst_truncate because either could return -1 due to fstat/ftruncate
> >>> failure.
> >>>
> >>> Fixes: 628e112afdd98c5 ("xfs_io: implement 'copy_range' command")
> >>> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> >>> Reviewed-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >>> Reviewed-by: Christoph Hellwig <hch@lst.de>
> >>
> >> I see this has reviews already, but I'm not conviced 1 is the right
> >> return on error.
> >>
> >> i.e. the error condition just prior returns 0:
> >>
> >>         fd = openfile(argv[optind], NULL, IO_READONLY, 0, NULL);
> >>         if (fd < 0)
> >>                 return 0;
> >>
> >> but OTOH if copy_file_range_cmd() fails we return nonzero.  Argh.
> >>
> >> Is there a rhyme or reason here that I'm missing?  Is it broken now
> >> so just do a coinflip on the new return for now?
> > 
> > Pretty much. :(
> > 
> > I don't really care if you change the return value, I just thought it
> > was bad form not to check the syscall/helper return values at all.
> 
> Yeah no argument there.  Ok, so this function already returns inconsistently,
> I'll worry about that later (I hope)

I think I've still got a patch sitting around that fixes the
inconsistent return values across most of xfs_io. Oh, I posted it at
one point, too.

https://marc.info/?l=linux-xfs&m=152644986309024&w=2

I guess I should resurrect it at some point.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[PATCH 08/36] xfs_io: statx -r should print attributes_mask

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

We're dumping the raw structure, so we ought to dump everything,
including the attributes_mask field.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 io/stat.c  |    4 ++++
 io/statx.h |    2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)


diff --git a/io/stat.c b/io/stat.c
index 64662b43..517be66e 100644
--- a/io/stat.c
+++ b/io/stat.c
@@ -6,6 +6,9 @@
  * Portions of statx support written by David Howells (dhowells@redhat.com)
  */
 
+/* Try to pick up statx definitions from the system headers. */
+#include <linux/stat.h>
+
 #include "command.h"
 #include "input.h"
 #include "init.h"
@@ -272,6 +275,7 @@ dump_raw_statx(struct statx *stx)
 	printf("stat.ino = %llu\n", (unsigned long long)stx->stx_ino);
 	printf("stat.size = %llu\n", (unsigned long long)stx->stx_size);
 	printf("stat.blocks = %llu\n", (unsigned long long)stx->stx_blocks);
+	printf("stat.attributes_mask = 0x%llx\n", (unsigned long long)stx->stx_attributes_mask);
 	printf("stat.atime.tv_sec = %lld\n", (long long)stx->stx_atime.tv_sec);
 	printf("stat.atime.tv_nsec = %d\n", stx->stx_atime.tv_nsec);
 	printf("stat.btime.tv_sec = %lld\n", (long long)stx->stx_btime.tv_sec);
diff --git a/io/statx.h b/io/statx.h
index 4e4b31ee..4f40eaa1 100644
--- a/io/statx.h
+++ b/io/statx.h
@@ -107,7 +107,7 @@ struct statx {
 	__u64	stx_ino;	/* Inode number */
 	__u64	stx_size;	/* File size */
 	__u64	stx_blocks;	/* Number of 512-byte blocks allocated */
-	__u64	__spare1[1];
+	__u64	stx_attributes_mask; /* Mask to show what's supported in stx_attributes */
 	/* 0x40 */
 	struct statx_timestamp	stx_atime;	/* Last access time */
 	struct statx_timestamp	stx_btime;	/* File creation time */

Re: [PATCH 08/36] xfs_io: statx -r should print attributes_mask

[Eric Sandeen]

On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> We're dumping the raw structure, so we ought to dump everything,
> including the attributes_mask field.

Ok - this was added to the kernel struct later, which is why we
didn't have it in the original userspace implementation, I guess:

commit 3209f68b3ca4667069923a325c88b21131bfdf9f
Author: David Howells <dhowells@redhat.com>
Date:   Fri Mar 31 18:32:17 2017 +0100

    statx: Include a mask for stx_attributes in struct statx
    
    Include a mask in struct stat to indicate which bits of stx_attributes the
    filesystem actually supports.
    
    This would also be useful if we add another system call that allows you to
    do a 'bulk attribute set' and pass in a statx struct with the masks
    appropriately set to say what you want to set.
    
    Signed-off-by: David Howells <dhowells@redhat.com>
    Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> ---
>  io/stat.c  |    4 ++++
>  io/statx.h |    2 +-
>  2 files changed, 5 insertions(+), 1 deletion(-)
> 
> 
> diff --git a/io/stat.c b/io/stat.c
> index 64662b43..517be66e 100644
> --- a/io/stat.c
> +++ b/io/stat.c
> @@ -6,6 +6,9 @@
>   * Portions of statx support written by David Howells (dhowells@redhat.com)
>   */
>  
> +/* Try to pick up statx definitions from the system headers. */
> +#include <linux/stat.h>
> +
>  #include "command.h"
>  #include "input.h"
>  #include "init.h"
> @@ -272,6 +275,7 @@ dump_raw_statx(struct statx *stx)
>  	printf("stat.ino = %llu\n", (unsigned long long)stx->stx_ino);
>  	printf("stat.size = %llu\n", (unsigned long long)stx->stx_size);
>  	printf("stat.blocks = %llu\n", (unsigned long long)stx->stx_blocks);
> +	printf("stat.attributes_mask = 0x%llx\n", (unsigned long long)stx->stx_attributes_mask);
>  	printf("stat.atime.tv_sec = %lld\n", (long long)stx->stx_atime.tv_sec);
>  	printf("stat.atime.tv_nsec = %d\n", stx->stx_atime.tv_nsec);
>  	printf("stat.btime.tv_sec = %lld\n", (long long)stx->stx_btime.tv_sec);
> diff --git a/io/statx.h b/io/statx.h
> index 4e4b31ee..4f40eaa1 100644
> --- a/io/statx.h
> +++ b/io/statx.h
> @@ -107,7 +107,7 @@ struct statx {
>  	__u64	stx_ino;	/* Inode number */
>  	__u64	stx_size;	/* File size */
>  	__u64	stx_blocks;	/* Number of 512-byte blocks allocated */
> -	__u64	__spare1[1];
> +	__u64	stx_attributes_mask; /* Mask to show what's supported in stx_attributes */
>  	/* 0x40 */
>  	struct statx_timestamp	stx_atime;	/* Last access time */
>  	struct statx_timestamp	stx_btime;	/* File creation time */
> 

[PATCH 09/36] xfs_io: don't walk off the end of argv in fzero_f

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

The fzero_f function doesn't check that there are enough non-switch
parameters to supply offset and length arguments to fallocate.  As a
result, we can walk off the end of the argv array and crash.  A
secondary problem is that we don't use getopt to detect the -k, which is
not how most xfs_io commands work.

Therefore, use getopt to detect the -k argument and rewire the offset
and length interpretation code to check optind and use argv correctly.
This bug is trivially reproduced by "xfs_io -c 'fzero -k 0' /some/file".

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 io/prealloc.c |   20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)


diff --git a/io/prealloc.c b/io/prealloc.c
index 9a372bae..6d452354 100644
--- a/io/prealloc.c
+++ b/io/prealloc.c
@@ -285,18 +285,24 @@ fzero_f(
 {
 	xfs_flock64_t	segment;
 	int		mode = FALLOC_FL_ZERO_RANGE;
-	int		index = 1;
+	int		c;
 
-	if (strncmp(argv[index], "-k", 3) == 0) {
-		mode |= FALLOC_FL_KEEP_SIZE;
-		index++;
+	while ((c = getopt(argc, argv, "k")) != EOF) {
+		switch (c) {
+		case 'k':
+			mode |= FALLOC_FL_KEEP_SIZE;
+			break;
+		default:
+			command_usage(&fzero_cmd);
+		}
 	}
+        if (optind != argc - 2)
+                return command_usage(&fzero_cmd);
 
-	if (!offset_length(argv[index], argv[index + 1], &segment))
+	if (!offset_length(argv[optind], argv[optind + 1], &segment))
 		return 0;
 
-	if (fallocate(file->fd, mode,
-			segment.l_start, segment.l_len)) {
+	if (fallocate(file->fd, mode, segment.l_start, segment.l_len)) {
 		perror("fallocate");
 		return 0;
 	}

[PATCH 09.5/36] xfs_io: document fzero_f -k option in manpage

[Eric Sandeen]

Perhaps the reason -k was broken is that nobody knew it existed?

Fixes: 938904c4 ("xfs_io: add fzero command for zeroing range via fallocate")
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
---

diff --git a/man/man8/xfs_io.8 b/man/man8/xfs_io.8
index fbf50df..980dcfd 100644
--- a/man/man8/xfs_io.8
+++ b/man/man8/xfs_io.8
@@ -516,10 +516,13 @@ Call fallocate with FALLOC_FL_UNSHARE_RANGE flag as described in the
 .BR fallocate (2)
 manual page to unshare all shared blocks within the range.
 .TP
-.BI fzero " offset length"
+.BI "fzero [ \-k ]" " offset length"
 Call fallocate with FALLOC_FL_ZERO_RANGE flag as described in the
 .BR fallocate (2)
 manual page to allocate and zero blocks within the range.
+With the
+.B -k
+option, use the FALLOC_FL_KEEP_SIZE flag as well.
 .TP
 .BI zero " offset length"
 Call xfsctl with

Re: [PATCH 09.5/36] xfs_io: document fzero_f -k option in manpage

[Darrick J. Wong]

On Thu, Mar 14, 2019 at 07:25:28PM -0500, Eric Sandeen wrote:
> Perhaps the reason -k was broken is that nobody knew it existed?
> 
> Fixes: 938904c4 ("xfs_io: add fzero command for zeroing range via fallocate")
> Signed-off-by: Eric Sandeen <sandeen@redhat.com>

Looks ok,
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>

--D

> ---
> 
> diff --git a/man/man8/xfs_io.8 b/man/man8/xfs_io.8
> index fbf50df..980dcfd 100644
> --- a/man/man8/xfs_io.8
> +++ b/man/man8/xfs_io.8
> @@ -516,10 +516,13 @@ Call fallocate with FALLOC_FL_UNSHARE_RANGE flag as described in the
>  .BR fallocate (2)
>  manual page to unshare all shared blocks within the range.
>  .TP
> -.BI fzero " offset length"
> +.BI "fzero [ \-k ]" " offset length"
>  Call fallocate with FALLOC_FL_ZERO_RANGE flag as described in the
>  .BR fallocate (2)
>  manual page to allocate and zero blocks within the range.
> +With the
> +.B -k
> +option, use the FALLOC_FL_KEEP_SIZE flag as well.
>  .TP
>  .BI zero " offset length"
>  Call xfsctl with
> 

Re: [PATCH 09/36] xfs_io: don't walk off the end of argv in fzero_f

[Eric Sandeen]

On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> The fzero_f function doesn't check that there are enough non-switch
> parameters to supply offset and length arguments to fallocate.  As a
> result, we can walk off the end of the argv array and crash.  A
> secondary problem is that we don't use getopt to detect the -k,

eek

>  which is
> not how most xfs_io commands work.

no it is not. :)

> 
> Therefore, use getopt to detect the -k argument and rewire the offset
> and length interpretation code to check optind and use argv correctly.
> This bug is trivially reproduced by "xfs_io -c 'fzero -k 0' /some/file".
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  io/prealloc.c |   20 +++++++++++++-------
>  1 file changed, 13 insertions(+), 7 deletions(-)
> 
> 
> diff --git a/io/prealloc.c b/io/prealloc.c
> index 9a372bae..6d452354 100644
> --- a/io/prealloc.c
> +++ b/io/prealloc.c
> @@ -285,18 +285,24 @@ fzero_f(
>  {
>  	xfs_flock64_t	segment;
>  	int		mode = FALLOC_FL_ZERO_RANGE;
> -	int		index = 1;
> +	int		c;
>  
> -	if (strncmp(argv[index], "-k", 3) == 0) {
> -		mode |= FALLOC_FL_KEEP_SIZE;
> -		index++;
> +	while ((c = getopt(argc, argv, "k")) != EOF) {
> +		switch (c) {
> +		case 'k':
> +			mode |= FALLOC_FL_KEEP_SIZE;
> +			break;
> +		default:
> +			command_usage(&fzero_cmd);
> +		}
>  	}
> +        if (optind != argc - 2)
> +                return command_usage(&fzero_cmd);
>  
> -	if (!offset_length(argv[index], argv[index + 1], &segment))
> +	if (!offset_length(argv[optind], argv[optind + 1], &segment))
>  		return 0;
>  
> -	if (fallocate(file->fd, mode,
> -			segment.l_start, segment.l_len)) {
> +	if (fallocate(file->fd, mode, segment.l_start, segment.l_len)) {
>  		perror("fallocate");
>  		return 0;
>  	}
> 

[PATCH 10/36] xfs_scrub_all: walk the lsblk device/fs hierarchy correctly

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Back when I was designing xfs_scrub_all, I naïvely assumed that the
emitted output would always list physical storage before the virtual
devices stacked atop it.  However, this is not actually true when one
omits the "NAME" column, which is crucial to forcing the output (json or
otherwise) to capture the block device hierarchy.  If the assumption is
violated, the program crashes with a python exception.

To fix this, force the hierarchal json output and restructure the
discovery routines to walk the json object that we receive, from the top
(physical devices) downwards to wherever there are live xfs filesystems.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/xfs_scrub_all.in |   28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)


diff --git a/scrub/xfs_scrub_all.in b/scrub/xfs_scrub_all.in
index c4e9899d..5b76b49a 100644
--- a/scrub/xfs_scrub_all.in
+++ b/scrub/xfs_scrub_all.in
@@ -28,9 +28,21 @@ def DEVNULL():
 
 def find_mounts():
 	'''Map mountpoints to physical disks.'''
+	def find_xfs_mounts(bdev, fs, lastdisk):
+		'''Attach lastdisk to each fs found under bdev.'''
+		if bdev['fstype'] == 'xfs' and bdev['mountpoint'] is not None:
+			mnt = bdev['mountpoint']
+			if mnt in fs:
+				fs[mnt].add(lastdisk)
+			else:
+				fs[mnt] = set([lastdisk])
+		if 'children' not in bdev:
+			return
+		for child in bdev['children']:
+			find_xfs_mounts(child, fs, lastdisk)
 
 	fs = {}
-	cmd=['lsblk', '-o', 'KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
+	cmd=['lsblk', '-o', 'NAME,KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
 	result = subprocess.Popen(cmd, stdout=subprocess.PIPE)
 	result.wait()
 	if result.returncode != 0:
@@ -38,18 +50,12 @@ def find_mounts():
 	sarray = [x.decode(sys.stdout.encoding) for x in result.stdout.readlines()]
 	output = ' '.join(sarray)
 	bdevdata = json.loads(output)
+
 	# The lsblk output had better be in disks-then-partitions order
 	for bdev in bdevdata['blockdevices']:
-		if bdev['type'] in ('disk', 'loop'):
-			lastdisk = bdev['kname']
-		if bdev['fstype'] == 'xfs':
-			mnt = bdev['mountpoint']
-			if mnt is None:
-				continue
-			if mnt in fs:
-				fs[mnt].add(lastdisk)
-			else:
-				fs[mnt] = set([lastdisk])
+		lastdisk = bdev['kname']
+		find_xfs_mounts(bdev, fs, lastdisk)
+
 	return fs
 
 def kill_systemd(unit, proc):

Re: [PATCH 10/36] xfs_scrub_all: walk the lsblk device/fs hierarchy correctly

[Eric Sandeen]

On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Back when I was designing xfs_scrub_all, I naïvely assumed that the
> emitted output would always list physical storage before the virtual
> devices stacked atop it.  However, this is not actually true when one
> omits the "NAME" column,

I think I asked this on IRC but for the record, is this incidental
or guaranteed by lsblk?  I don't see this distinction in the manpage.

I worry about depending on undocumented behavior...

Looking back through IRC, I think we shrugged over this, I asked kzak
but didn't get a reply, will try to find him again.

(also FWIW, older lsblk rejects -J, do we care?)

-Eric

> which is crucial to forcing the output (json or
> otherwise) to capture the block device hierarchy.  If the assumption is
> violated, the program crashes with a python exception.
> 
> To fix this, force the hierarchal json output and restructure the
> discovery routines to walk the json object that we receive, from the top
> (physical devices) downwards to wherever there are live xfs filesystems.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  scrub/xfs_scrub_all.in |   28 +++++++++++++++++-----------
>  1 file changed, 17 insertions(+), 11 deletions(-)
> 
> 
> diff --git a/scrub/xfs_scrub_all.in b/scrub/xfs_scrub_all.in
> index c4e9899d..5b76b49a 100644
> --- a/scrub/xfs_scrub_all.in
> +++ b/scrub/xfs_scrub_all.in
> @@ -28,9 +28,21 @@ def DEVNULL():
>  
>  def find_mounts():
>  	'''Map mountpoints to physical disks.'''
> +	def find_xfs_mounts(bdev, fs, lastdisk):
> +		'''Attach lastdisk to each fs found under bdev.'''
> +		if bdev['fstype'] == 'xfs' and bdev['mountpoint'] is not None:
> +			mnt = bdev['mountpoint']
> +			if mnt in fs:
> +				fs[mnt].add(lastdisk)
> +			else:
> +				fs[mnt] = set([lastdisk])
> +		if 'children' not in bdev:
> +			return
> +		for child in bdev['children']:
> +			find_xfs_mounts(child, fs, lastdisk)
>  
>  	fs = {}
> -	cmd=['lsblk', '-o', 'KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
> +	cmd=['lsblk', '-o', 'NAME,KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
>  	result = subprocess.Popen(cmd, stdout=subprocess.PIPE)
>  	result.wait()
>  	if result.returncode != 0:
> @@ -38,18 +50,12 @@ def find_mounts():
>  	sarray = [x.decode(sys.stdout.encoding) for x in result.stdout.readlines()]
>  	output = ' '.join(sarray)
>  	bdevdata = json.loads(output)
> +
>  	# The lsblk output had better be in disks-then-partitions order
>  	for bdev in bdevdata['blockdevices']:
> -		if bdev['type'] in ('disk', 'loop'):
> -			lastdisk = bdev['kname']
> -		if bdev['fstype'] == 'xfs':
> -			mnt = bdev['mountpoint']
> -			if mnt is None:
> -				continue
> -			if mnt in fs:
> -				fs[mnt].add(lastdisk)
> -			else:
> -				fs[mnt] = set([lastdisk])
> +		lastdisk = bdev['kname']
> +		find_xfs_mounts(bdev, fs, lastdisk)
> +
>  	return fs
>  
>  def kill_systemd(unit, proc):
> 

Re: [PATCH 10/36] xfs_scrub_all: walk the lsblk device/fs hierarchy correctly

[Darrick J. Wong]

On Thu, Mar 14, 2019 at 09:46:26PM -0500, Eric Sandeen wrote:
> On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Back when I was designing xfs_scrub_all, I naïvely assumed that the
> > emitted output would always list physical storage before the virtual
> > devices stacked atop it.  However, this is not actually true when one
> > omits the "NAME" column,
> 
> I think I asked this on IRC but for the record, is this incidental
> or guaranteed by lsblk?  I don't see this distinction in the manpage.

Incidental.

> I worry about depending on undocumented behavior...
> 
> Looking back through IRC, I think we shrugged over this, I asked kzak
> but didn't get a reply, will try to find him again.

We did shrug about this.

> (also FWIW, older lsblk rejects -J, do we care?)

No, because xfs_scrub_all will just exit if lsblk returns nonzero.

--D

> -Eric
> 
> > which is crucial to forcing the output (json or
> > otherwise) to capture the block device hierarchy.  If the assumption is
> > violated, the program crashes with a python exception.
> > 
> > To fix this, force the hierarchal json output and restructure the
> > discovery routines to walk the json object that we receive, from the top
> > (physical devices) downwards to wherever there are live xfs filesystems.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  scrub/xfs_scrub_all.in |   28 +++++++++++++++++-----------
> >  1 file changed, 17 insertions(+), 11 deletions(-)
> > 
> > 
> > diff --git a/scrub/xfs_scrub_all.in b/scrub/xfs_scrub_all.in
> > index c4e9899d..5b76b49a 100644
> > --- a/scrub/xfs_scrub_all.in
> > +++ b/scrub/xfs_scrub_all.in
> > @@ -28,9 +28,21 @@ def DEVNULL():
> >  
> >  def find_mounts():
> >  	'''Map mountpoints to physical disks.'''
> > +	def find_xfs_mounts(bdev, fs, lastdisk):
> > +		'''Attach lastdisk to each fs found under bdev.'''
> > +		if bdev['fstype'] == 'xfs' and bdev['mountpoint'] is not None:
> > +			mnt = bdev['mountpoint']
> > +			if mnt in fs:
> > +				fs[mnt].add(lastdisk)
> > +			else:
> > +				fs[mnt] = set([lastdisk])
> > +		if 'children' not in bdev:
> > +			return
> > +		for child in bdev['children']:
> > +			find_xfs_mounts(child, fs, lastdisk)
> >  
> >  	fs = {}
> > -	cmd=['lsblk', '-o', 'KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
> > +	cmd=['lsblk', '-o', 'NAME,KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
> >  	result = subprocess.Popen(cmd, stdout=subprocess.PIPE)
> >  	result.wait()
> >  	if result.returncode != 0:
> > @@ -38,18 +50,12 @@ def find_mounts():
> >  	sarray = [x.decode(sys.stdout.encoding) for x in result.stdout.readlines()]
> >  	output = ' '.join(sarray)
> >  	bdevdata = json.loads(output)
> > +
> >  	# The lsblk output had better be in disks-then-partitions order
> >  	for bdev in bdevdata['blockdevices']:
> > -		if bdev['type'] in ('disk', 'loop'):
> > -			lastdisk = bdev['kname']
> > -		if bdev['fstype'] == 'xfs':
> > -			mnt = bdev['mountpoint']
> > -			if mnt is None:
> > -				continue
> > -			if mnt in fs:
> > -				fs[mnt].add(lastdisk)
> > -			else:
> > -				fs[mnt] = set([lastdisk])
> > +		lastdisk = bdev['kname']
> > +		find_xfs_mounts(bdev, fs, lastdisk)
> > +
> >  	return fs
> >  
> >  def kill_systemd(unit, proc):
> > 

[PATCH 11/36] xfs_scrub_all.timer: activate after most of the system is up

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

We really don't want the xfs_scrub_all timer triggering while the system
is booting up because not all the mounts will have finished, networking
might not be up for reporting, and slowing down bootup annoys people.
Therefore, delay the xfs_scrub_all service's activation until after the
system has started all the big pieces it's going to start.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/xfs_scrub_all.service.in |    1 +
 1 file changed, 1 insertion(+)


diff --git a/scrub/xfs_scrub_all.service.in b/scrub/xfs_scrub_all.service.in
index 66f82fc7..b1b80da4 100644
--- a/scrub/xfs_scrub_all.service.in
+++ b/scrub/xfs_scrub_all.service.in
@@ -2,6 +2,7 @@
 Description=Online XFS Metadata Check for All Filesystems
 ConditionACPower=true
 Documentation=man:xfs_scrub_all(8)
+After=paths.target multi-user.target network.target network-online.target systemd-networkd.service NetworkManager.service connman.service
 
 [Service]
 Type=oneshot

Re: [PATCH 11/36] xfs_scrub_all.timer: activate after most of the system is up

[Eric Sandeen]

On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> We really don't want the xfs_scrub_all timer triggering while the system
> is booting up because not all the mounts will have finished, networking
> might not be up for reporting, and slowing down bootup annoys people.
> Therefore, delay the xfs_scrub_all service's activation until after the
> system has started all the big pieces it's going to start.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

<mumble handwave systemd mumble>

what happens if you state an After requirement for something that doesn't
exist on the box?  Is it just ignored?

(he asks, after looking for what connman.service is and finds that it's not
ubiquitous ...)

> ---
>  scrub/xfs_scrub_all.service.in |    1 +
>  1 file changed, 1 insertion(+)
> 
> 
> diff --git a/scrub/xfs_scrub_all.service.in b/scrub/xfs_scrub_all.service.in
> index 66f82fc7..b1b80da4 100644
> --- a/scrub/xfs_scrub_all.service.in
> +++ b/scrub/xfs_scrub_all.service.in
> @@ -2,6 +2,7 @@
>  Description=Online XFS Metadata Check for All Filesystems
>  ConditionACPower=true
>  Documentation=man:xfs_scrub_all(8)
> +After=paths.target multi-user.target network.target network-online.target systemd-networkd.service NetworkManager.service connman.service
>  
>  [Service]
>  Type=oneshot
> 

Re: [PATCH 11/36] xfs_scrub_all.timer: activate after most of the system is up

[Darrick J. Wong]

On Thu, Mar 14, 2019 at 09:56:35PM -0500, Eric Sandeen wrote:
> 
> 
> On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > We really don't want the xfs_scrub_all timer triggering while the system
> > is booting up because not all the mounts will have finished, networking
> > might not be up for reporting, and slowing down bootup annoys people.
> > Therefore, delay the xfs_scrub_all service's activation until after the
> > system has started all the big pieces it's going to start.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> <mumble handwave systemd mumble>
> 
> what happens if you state an After requirement for something that doesn't
> exist on the box?  Is it just ignored?

Yes, it's ignored, like everything else systemd's service file parser
doesn't understand FBOFW.

--D

> (he asks, after looking for what connman.service is and finds that it's not
> ubiquitous ...)
> 
> > ---
> >  scrub/xfs_scrub_all.service.in |    1 +
> >  1 file changed, 1 insertion(+)
> > 
> > 
> > diff --git a/scrub/xfs_scrub_all.service.in b/scrub/xfs_scrub_all.service.in
> > index 66f82fc7..b1b80da4 100644
> > --- a/scrub/xfs_scrub_all.service.in
> > +++ b/scrub/xfs_scrub_all.service.in
> > @@ -2,6 +2,7 @@
> >  Description=Online XFS Metadata Check for All Filesystems
> >  ConditionACPower=true
> >  Documentation=man:xfs_scrub_all(8)
> > +After=paths.target multi-user.target network.target network-online.target systemd-networkd.service NetworkManager.service connman.service
> >  
> >  [Service]
> >  Type=oneshot
> > 

Re: [PATCH 11/36] xfs_scrub_all.timer: activate after most of the system is up

[Eric Sandeen]

On 3/14/19 4:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> We really don't want the xfs_scrub_all timer triggering while the system
> is booting up because not all the mounts will have finished, networking
> might not be up for reporting, and slowing down bootup annoys people.
> Therefore, delay the xfs_scrub_all service's activation until after the
> system has started all the big pieces it's going to start.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/xfs_scrub_all.service.in |    1 +
>  1 file changed, 1 insertion(+)
> 
> 
> diff --git a/scrub/xfs_scrub_all.service.in b/scrub/xfs_scrub_all.service.in
> index 66f82fc7..b1b80da4 100644
> --- a/scrub/xfs_scrub_all.service.in
> +++ b/scrub/xfs_scrub_all.service.in
> @@ -2,6 +2,7 @@
>  Description=Online XFS Metadata Check for All Filesystems
>  ConditionACPower=true
>  Documentation=man:xfs_scrub_all(8)
> +After=paths.target multi-user.target network.target network-online.target systemd-networkd.service NetworkManager.service connman.service
>  
>  [Service]
>  Type=oneshot
> 

[PATCH 12/36] xfs_scrub: rename the global nr_threads

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Various functions have nr_threads local variables that shadow the global
one.  Since the global one forces the number of threads we use, change
its name to remove this ambiguity and reflect what it really does.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/common.c    |    4 ++--
 scrub/disk.c      |    4 ++--
 scrub/xfs_scrub.c |    6 +++---
 scrub/xfs_scrub.h |    2 +-
 4 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/scrub/common.c b/scrub/common.c
index 78afc4bf..c877c7c8 100644
--- a/scrub/common.c
+++ b/scrub/common.c
@@ -231,8 +231,8 @@ unsigned int
 scrub_nproc(
 	struct scrub_ctx	*ctx)
 {
-	if (nr_threads)
-		return nr_threads;
+	if (force_nr_threads)
+		return force_nr_threads;
 	return ctx->nr_io_threads;
 }
 
diff --git a/scrub/disk.c b/scrub/disk.c
index 7daa508e..dd109533 100644
--- a/scrub/disk.c
+++ b/scrub/disk.c
@@ -75,8 +75,8 @@ unsigned int
 disk_heads(
 	struct disk		*disk)
 {
-	if (nr_threads)
-		return nr_threads;
+	if (force_nr_threads)
+		return force_nr_threads;
 	return __disk_heads(disk);
 }
 
diff --git a/scrub/xfs_scrub.c b/scrub/xfs_scrub.c
index b8138000..71fc274f 100644
--- a/scrub/xfs_scrub.c
+++ b/scrub/xfs_scrub.c
@@ -133,7 +133,7 @@ unsigned int			bg_mode;
 int				nproc;
 
 /* Number of threads we're allowed to use. */
-unsigned int			nr_threads;
+unsigned int			force_nr_threads;
 
 /* Verbosity; higher values print more information. */
 bool				verbose;
@@ -589,7 +589,7 @@ main(
 			}
 			break;
 		case 'b':
-			nr_threads = 1;
+			force_nr_threads = 1;
 			bg_mode++;
 			break;
 		case 'C':
@@ -659,7 +659,7 @@ main(
 			perror("nr_threads");
 			usage();
 		}
-		nr_threads = x;
+		force_nr_threads = x;
 	}
 
 	if (optind != argc - 1)
diff --git a/scrub/xfs_scrub.h b/scrub/xfs_scrub.h
index a961d8fd..a459e4b5 100644
--- a/scrub/xfs_scrub.h
+++ b/scrub/xfs_scrub.h
@@ -10,7 +10,7 @@ extern char *progname;
 
 #define _PATH_PROC_MOUNTS	"/proc/mounts"
 
-extern unsigned int		nr_threads;
+extern unsigned int		force_nr_threads;
 extern unsigned int		bg_mode;
 extern unsigned int		debug;
 extern int			nproc;

Re: [PATCH 12/36] xfs_scrub: rename the global nr_threads

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Various functions have nr_threads local variables that shadow the global
> one.  Since the global one forces the number of threads we use, change
> its name to remove this ambiguity and reflect what it really does.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/common.c    |    4 ++--
>  scrub/disk.c      |    4 ++--
>  scrub/xfs_scrub.c |    6 +++---
>  scrub/xfs_scrub.h |    2 +-
>  4 files changed, 8 insertions(+), 8 deletions(-)
> 
> 
> diff --git a/scrub/common.c b/scrub/common.c
> index 78afc4bf..c877c7c8 100644
> --- a/scrub/common.c
> +++ b/scrub/common.c
> @@ -231,8 +231,8 @@ unsigned int
>  scrub_nproc(
>  	struct scrub_ctx	*ctx)
>  {
> -	if (nr_threads)
> -		return nr_threads;
> +	if (force_nr_threads)
> +		return force_nr_threads;
>  	return ctx->nr_io_threads;
>  }
>  
> diff --git a/scrub/disk.c b/scrub/disk.c
> index 7daa508e..dd109533 100644
> --- a/scrub/disk.c
> +++ b/scrub/disk.c
> @@ -75,8 +75,8 @@ unsigned int
>  disk_heads(
>  	struct disk		*disk)
>  {
> -	if (nr_threads)
> -		return nr_threads;
> +	if (force_nr_threads)
> +		return force_nr_threads;
>  	return __disk_heads(disk);
>  }
>  
> diff --git a/scrub/xfs_scrub.c b/scrub/xfs_scrub.c
> index b8138000..71fc274f 100644
> --- a/scrub/xfs_scrub.c
> +++ b/scrub/xfs_scrub.c
> @@ -133,7 +133,7 @@ unsigned int			bg_mode;
>  int				nproc;
>  
>  /* Number of threads we're allowed to use. */
> -unsigned int			nr_threads;
> +unsigned int			force_nr_threads;
>  
>  /* Verbosity; higher values print more information. */
>  bool				verbose;
> @@ -589,7 +589,7 @@ main(
>  			}
>  			break;
>  		case 'b':
> -			nr_threads = 1;
> +			force_nr_threads = 1;
>  			bg_mode++;
>  			break;
>  		case 'C':
> @@ -659,7 +659,7 @@ main(
>  			perror("nr_threads");
>  			usage();
>  		}
> -		nr_threads = x;
> +		force_nr_threads = x;
>  	}
>  
>  	if (optind != argc - 1)
> diff --git a/scrub/xfs_scrub.h b/scrub/xfs_scrub.h
> index a961d8fd..a459e4b5 100644
> --- a/scrub/xfs_scrub.h
> +++ b/scrub/xfs_scrub.h
> @@ -10,7 +10,7 @@ extern char *progname;
>  
>  #define _PATH_PROC_MOUNTS	"/proc/mounts"
>  
> -extern unsigned int		nr_threads;
> +extern unsigned int		force_nr_threads;
>  extern unsigned int		bg_mode;
>  extern unsigned int		debug;
>  extern int			nproc;
> 

[PATCH 13/36] xfs_scrub: use datadev parallelization estimates for thread count

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

During phases 2-5, xfs_scrub should estimate the level of
parallelization possible on the data device to determine the number of
threads spawned to scrub filesystem metadata, not just blindly using the
number of CPUs.  This avoids flooding non-rotational storage with random
reads, which totally destroys performance and makes scrub runtimes
higher.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase1.c |   14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)


diff --git a/scrub/phase1.c b/scrub/phase1.c
index 2113014b..6b472147 100644
--- a/scrub/phase1.c
+++ b/scrub/phase1.c
@@ -109,13 +109,6 @@ _("Must be root to run scrub."));
 		return false;
 	}
 
-	ctx->nr_io_threads = nproc;
-	if (verbose) {
-		fprintf(stdout, _("%s: using %d threads to scrub.\n"),
-				ctx->mntpoint, scrub_nproc(ctx));
-		fflush(stdout);
-	}
-
 	if (!platform_test_xfs_fd(ctx->mnt_fd)) {
 		str_info(ctx, ctx->mntpoint,
 _("Does not appear to be an XFS filesystem!"));
@@ -193,6 +186,13 @@ _("Unable to find realtime device path."));
 		return false;
 	}
 
+	ctx->nr_io_threads = disk_heads(ctx->datadev);
+	if (verbose) {
+		fprintf(stdout, _("%s: using %d threads to scrub.\n"),
+				ctx->mntpoint, scrub_nproc(ctx));
+		fflush(stdout);
+	}
+
 	if (ctx->fsinfo.fs_log) {
 		ctx->logdev = disk_open(ctx->fsinfo.fs_log);
 		if (error) {

Re: [PATCH 13/36] xfs_scrub: use datadev parallelization estimates for thread count

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> During phases 2-5, xfs_scrub should estimate the level of
> parallelization possible on the data device to determine the number of
> threads spawned to scrub filesystem metadata, not just blindly using the
> number of CPUs.  This avoids flooding non-rotational storage with random
> reads, which totally destroys performance and makes scrub runtimes
> higher.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

seems like when I first looked at this I had a concern but now I don't
remember, and today it seems fine? :)

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/phase1.c |   14 +++++++-------
>  1 file changed, 7 insertions(+), 7 deletions(-)
> 
> 
> diff --git a/scrub/phase1.c b/scrub/phase1.c
> index 2113014b..6b472147 100644
> --- a/scrub/phase1.c
> +++ b/scrub/phase1.c
> @@ -109,13 +109,6 @@ _("Must be root to run scrub."));
>  		return false;
>  	}
>  
> -	ctx->nr_io_threads = nproc;
> -	if (verbose) {
> -		fprintf(stdout, _("%s: using %d threads to scrub.\n"),
> -				ctx->mntpoint, scrub_nproc(ctx));
> -		fflush(stdout);
> -	}
> -
>  	if (!platform_test_xfs_fd(ctx->mnt_fd)) {
>  		str_info(ctx, ctx->mntpoint,
>  _("Does not appear to be an XFS filesystem!"));
> @@ -193,6 +186,13 @@ _("Unable to find realtime device path."));
>  		return false;
>  	}
>  
> +	ctx->nr_io_threads = disk_heads(ctx->datadev);
> +	if (verbose) {
> +		fprintf(stdout, _("%s: using %d threads to scrub.\n"),
> +				ctx->mntpoint, scrub_nproc(ctx));
> +		fflush(stdout);
> +	}
> +
>  	if (ctx->fsinfo.fs_log) {
>  		ctx->logdev = disk_open(ctx->fsinfo.fs_log);
>  		if (error) {
> 

Re: [PATCH 13/36] xfs_scrub: use datadev parallelization estimates for thread count

[Darrick J. Wong]

On Fri, Mar 15, 2019 at 12:36:34PM -0500, Eric Sandeen wrote:
> On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > During phases 2-5, xfs_scrub should estimate the level of
> > parallelization possible on the data device to determine the number of
> > threads spawned to scrub filesystem metadata, not just blindly using the
> > number of CPUs.  This avoids flooding non-rotational storage with random

s/non-//g in the line above, is what you complained about last time and
then I didn't bother to fix for subsequent repostings. :(

--D

> > reads, which totally destroys performance and makes scrub runtimes
> > higher.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> seems like when I first looked at this I had a concern but now I don't
> remember, and today it seems fine? :)
> 
> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
> 
> > ---
> >  scrub/phase1.c |   14 +++++++-------
> >  1 file changed, 7 insertions(+), 7 deletions(-)
> > 
> > 
> > diff --git a/scrub/phase1.c b/scrub/phase1.c
> > index 2113014b..6b472147 100644
> > --- a/scrub/phase1.c
> > +++ b/scrub/phase1.c
> > @@ -109,13 +109,6 @@ _("Must be root to run scrub."));
> >  		return false;
> >  	}
> >  
> > -	ctx->nr_io_threads = nproc;
> > -	if (verbose) {
> > -		fprintf(stdout, _("%s: using %d threads to scrub.\n"),
> > -				ctx->mntpoint, scrub_nproc(ctx));
> > -		fflush(stdout);
> > -	}
> > -
> >  	if (!platform_test_xfs_fd(ctx->mnt_fd)) {
> >  		str_info(ctx, ctx->mntpoint,
> >  _("Does not appear to be an XFS filesystem!"));
> > @@ -193,6 +186,13 @@ _("Unable to find realtime device path."));
> >  		return false;
> >  	}
> >  
> > +	ctx->nr_io_threads = disk_heads(ctx->datadev);
> > +	if (verbose) {
> > +		fprintf(stdout, _("%s: using %d threads to scrub.\n"),
> > +				ctx->mntpoint, scrub_nproc(ctx));
> > +		fflush(stdout);
> > +	}
> > +
> >  	if (ctx->fsinfo.fs_log) {
> >  		ctx->logdev = disk_open(ctx->fsinfo.fs_log);
> >  		if (error) {
> > 

[PATCH 14/36] xfs_scrub: don't expose internal pool state

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

In xfs_scrub, the read/verify pool tries to coalesce the media
verification requests into a smaller number of large IOs.  There's no
need to force callers to keep track of this internal state, so just move
all that into read_verify.c.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase6.c      |   23 +++++++----------------
 scrub/read_verify.c |   40 ++++++++++++++++++++++++++++++++++------
 scrub/read_verify.h |   16 ++++------------
 3 files changed, 45 insertions(+), 34 deletions(-)


diff --git a/scrub/phase6.c b/scrub/phase6.c
index ead48d77..fe121769 100644
--- a/scrub/phase6.c
+++ b/scrub/phase6.c
@@ -9,7 +9,6 @@
 #include <sys/statvfs.h>
 #include "handle.h"
 #include "path.h"
-#include "ptvar.h"
 #include "workqueue.h"
 #include "xfs_scrub.h"
 #include "common.h"
@@ -290,7 +289,6 @@ xfs_report_verify_errors(
 
 struct xfs_verify_extent {
 	struct read_verify_pool	*readverify;
-	struct ptvar		*rvstate;
 	struct bitmap		*d_bad;		/* bytes */
 	struct bitmap		*r_bad;		/* bytes */
 };
@@ -424,13 +422,13 @@ xfs_check_rmap(
 	/* Schedule the read verify command for (eventual) running. */
 	disk = xfs_dev_to_disk(ctx, map->fmr_device);
 
-	read_verify_schedule_io(ve->readverify, ptvar_get(ve->rvstate), disk,
-			map->fmr_physical, map->fmr_length, ve);
+	read_verify_schedule_io(ve->readverify, disk, map->fmr_physical,
+			map->fmr_length, ve);
 
 out:
 	/* Is this the last extent?  Fire off the read. */
 	if (map->fmr_flags & FMR_OF_LAST)
-		read_verify_force_io(ve->readverify, ptvar_get(ve->rvstate));
+		read_verify_force_io(ve->readverify);
 
 	return true;
 }
@@ -450,16 +448,10 @@ xfs_scan_blocks(
 	struct xfs_verify_extent	ve;
 	bool				moveon;
 
-	ve.rvstate = ptvar_init(scrub_nproc(ctx), sizeof(struct read_verify));
-	if (!ve.rvstate) {
-		str_errno(ctx, ctx->mntpoint);
-		return false;
-	}
-
 	moveon = bitmap_init(&ve.d_bad);
 	if (!moveon) {
 		str_errno(ctx, ctx->mntpoint);
-		goto out_ve;
+		goto out;
 	}
 
 	moveon = bitmap_init(&ve.r_bad);
@@ -469,7 +461,8 @@ xfs_scan_blocks(
 	}
 
 	ve.readverify = read_verify_pool_init(ctx, ctx->geo.blocksize,
-			xfs_check_rmap_ioerr, disk_heads(ctx->datadev));
+			xfs_check_rmap_ioerr, disk_heads(ctx->datadev),
+			scrub_nproc(ctx));
 	if (!ve.readverify) {
 		moveon = false;
 		str_info(ctx, ctx->mntpoint,
@@ -489,7 +482,6 @@ _("Could not create media verifier."));
 
 	bitmap_free(&ve.r_bad);
 	bitmap_free(&ve.d_bad);
-	ptvar_free(ve.rvstate);
 	return moveon;
 
 out_pool:
@@ -498,8 +490,7 @@ _("Could not create media verifier."));
 	bitmap_free(&ve.r_bad);
 out_dbad:
 	bitmap_free(&ve.d_bad);
-out_ve:
-	ptvar_free(ve.rvstate);
+out:
 	return moveon;
 }
 
diff --git a/scrub/read_verify.c b/scrub/read_verify.c
index 75cb53ca..b5774736 100644
--- a/scrub/read_verify.c
+++ b/scrub/read_verify.c
@@ -7,6 +7,7 @@
 #include <stdint.h>
 #include <stdlib.h>
 #include <sys/statvfs.h>
+#include "ptvar.h"
 #include "workqueue.h"
 #include "path.h"
 #include "xfs_scrub.h"
@@ -36,22 +37,40 @@
 /* Tolerate 64k holes in adjacent read verify requests. */
 #define RVP_IO_BATCH_LOCALITY	(65536)
 
+struct read_verify {
+	void			*io_end_arg;
+	struct disk		*io_disk;
+	uint64_t		io_start;	/* bytes */
+	uint64_t		io_length;	/* bytes */
+};
+
 struct read_verify_pool {
 	struct workqueue	wq;		/* thread pool */
 	struct scrub_ctx	*ctx;		/* scrub context */
 	void			*readbuf;	/* read buffer */
 	struct ptcounter	*verified_bytes;
+	struct ptvar		*rvstate;	/* combines read requests */
 	read_verify_ioerr_fn_t	ioerr_fn;	/* io error callback */
 	size_t			miniosz;	/* minimum io size, bytes */
 };
 
-/* Create a thread pool to run read verifiers. */
+/*
+ * Create a thread pool to run read verifiers.
+ *
+ * @miniosz is the minimum size of an IO to expect (in bytes).
+ * @ioerr_fn will be called when IO errors occur.
+ * @nproc is the maximum number of verify requests that may be sent to a disk
+ * at any given time.
+ * @submitter_threads is the number of threads that may be sending verify
+ * requests at any given time.
+ */
 struct read_verify_pool *
 read_verify_pool_init(
 	struct scrub_ctx		*ctx,
 	size_t				miniosz,
 	read_verify_ioerr_fn_t		ioerr_fn,
-	unsigned int			nproc)
+	unsigned int			nproc,
+	unsigned int			submitter_threads)
 {
 	struct read_verify_pool		*rvp;
 	bool				ret;
@@ -71,14 +90,20 @@ read_verify_pool_init(
 	rvp->miniosz = miniosz;
 	rvp->ctx = ctx;
 	rvp->ioerr_fn = ioerr_fn;
+	rvp->rvstate = ptvar_init(submitter_threads,
+			sizeof(struct read_verify));
+	if (rvp->rvstate == NULL)
+		goto out_counter;
 	/* Run in the main thread if we only want one thread. */
 	if (nproc == 1)
 		nproc = 0;
 	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp, nproc);
 	if (ret)
-		goto out_counter;
+		goto out_rvstate;
 	return rvp;
 
+out_rvstate:
+	ptvar_free(rvp->rvstate);
 out_counter:
 	ptcounter_free(rvp->verified_bytes);
 out_buf:
@@ -101,6 +126,7 @@ void
 read_verify_pool_destroy(
 	struct read_verify_pool		*rvp)
 {
+	ptvar_free(rvp->rvstate);
 	ptcounter_free(rvp->verified_bytes);
 	free(rvp->readbuf);
 	free(rvp);
@@ -186,16 +212,17 @@ _("Could not queue read-verify work."));
 bool
 read_verify_schedule_io(
 	struct read_verify_pool		*rvp,
-	struct read_verify		*rv,
 	struct disk			*disk,
 	uint64_t			start,
 	uint64_t			length,
 	void				*end_arg)
 {
+	struct read_verify		*rv;
 	uint64_t			req_end;
 	uint64_t			rv_end;
 
 	assert(rvp->readbuf);
+	rv = ptvar_get(rvp->rvstate);
 	req_end = start + length;
 	rv_end = rv->io_start + rv->io_length;
 
@@ -229,12 +256,13 @@ read_verify_schedule_io(
 /* Force any stashed IOs into the verifier. */
 bool
 read_verify_force_io(
-	struct read_verify_pool		*rvp,
-	struct read_verify		*rv)
+	struct read_verify_pool		*rvp)
 {
+	struct read_verify		*rv;
 	bool				moveon;
 
 	assert(rvp->readbuf);
+	rv = ptvar_get(rvp->rvstate);
 	if (rv->io_length == 0)
 		return true;
 
diff --git a/scrub/read_verify.h b/scrub/read_verify.h
index 38f1cd1a..1e7fd83f 100644
--- a/scrub/read_verify.h
+++ b/scrub/read_verify.h
@@ -16,21 +16,13 @@ typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
 
 struct read_verify_pool *read_verify_pool_init(struct scrub_ctx *ctx,
 		size_t miniosz, read_verify_ioerr_fn_t ioerr_fn,
-		unsigned int nproc);
+		unsigned int nproc, unsigned int submitter_threads);
 void read_verify_pool_flush(struct read_verify_pool *rvp);
 void read_verify_pool_destroy(struct read_verify_pool *rvp);
 
-struct read_verify {
-	void			*io_end_arg;
-	struct disk		*io_disk;
-	uint64_t		io_start;	/* bytes */
-	uint64_t		io_length;	/* bytes */
-};
-
-bool read_verify_schedule_io(struct read_verify_pool *rvp,
-		struct read_verify *rv, struct disk *disk, uint64_t start,
-		uint64_t length, void *end_arg);
-bool read_verify_force_io(struct read_verify_pool *rvp, struct read_verify *rv);
+bool read_verify_schedule_io(struct read_verify_pool *rvp, struct disk *disk,
+		uint64_t start, uint64_t length, void *end_arg);
+bool read_verify_force_io(struct read_verify_pool *rvp);
 uint64_t read_verify_bytes(struct read_verify_pool *rvp);
 
 #endif /* XFS_SCRUB_READ_VERIFY_H_ */

Re: [PATCH 14/36] xfs_scrub: don't expose internal pool state

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> In xfs_scrub, the read/verify pool tries to coalesce the media
> verification requests into a smaller number of large IOs.  There's no
> need to force callers to keep track of this internal state, so just move
> all that into read_verify.c.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/phase6.c      |   23 +++++++----------------
>  scrub/read_verify.c |   40 ++++++++++++++++++++++++++++++++++------
>  scrub/read_verify.h |   16 ++++------------
>  3 files changed, 45 insertions(+), 34 deletions(-)
> 
> 
> diff --git a/scrub/phase6.c b/scrub/phase6.c
> index ead48d77..fe121769 100644
> --- a/scrub/phase6.c
> +++ b/scrub/phase6.c
> @@ -9,7 +9,6 @@
>  #include <sys/statvfs.h>
>  #include "handle.h"
>  #include "path.h"
> -#include "ptvar.h"
>  #include "workqueue.h"
>  #include "xfs_scrub.h"
>  #include "common.h"
> @@ -290,7 +289,6 @@ xfs_report_verify_errors(
>  
>  struct xfs_verify_extent {
>  	struct read_verify_pool	*readverify;
> -	struct ptvar		*rvstate;
>  	struct bitmap		*d_bad;		/* bytes */
>  	struct bitmap		*r_bad;		/* bytes */
>  };
> @@ -424,13 +422,13 @@ xfs_check_rmap(
>  	/* Schedule the read verify command for (eventual) running. */
>  	disk = xfs_dev_to_disk(ctx, map->fmr_device);
>  
> -	read_verify_schedule_io(ve->readverify, ptvar_get(ve->rvstate), disk,
> -			map->fmr_physical, map->fmr_length, ve);
> +	read_verify_schedule_io(ve->readverify, disk, map->fmr_physical,
> +			map->fmr_length, ve);
>  
>  out:
>  	/* Is this the last extent?  Fire off the read. */
>  	if (map->fmr_flags & FMR_OF_LAST)
> -		read_verify_force_io(ve->readverify, ptvar_get(ve->rvstate));
> +		read_verify_force_io(ve->readverify);
>  
>  	return true;
>  }
> @@ -450,16 +448,10 @@ xfs_scan_blocks(
>  	struct xfs_verify_extent	ve;
>  	bool				moveon;
>  
> -	ve.rvstate = ptvar_init(scrub_nproc(ctx), sizeof(struct read_verify));
> -	if (!ve.rvstate) {
> -		str_errno(ctx, ctx->mntpoint);
> -		return false;
> -	}
> -
>  	moveon = bitmap_init(&ve.d_bad);
>  	if (!moveon) {
>  		str_errno(ctx, ctx->mntpoint);
> -		goto out_ve;
> +		goto out;
>  	}
>  
>  	moveon = bitmap_init(&ve.r_bad);
> @@ -469,7 +461,8 @@ xfs_scan_blocks(
>  	}
>  
>  	ve.readverify = read_verify_pool_init(ctx, ctx->geo.blocksize,
> -			xfs_check_rmap_ioerr, disk_heads(ctx->datadev));
> +			xfs_check_rmap_ioerr, disk_heads(ctx->datadev),
> +			scrub_nproc(ctx));
>  	if (!ve.readverify) {
>  		moveon = false;
>  		str_info(ctx, ctx->mntpoint,
> @@ -489,7 +482,6 @@ _("Could not create media verifier."));
>  
>  	bitmap_free(&ve.r_bad);
>  	bitmap_free(&ve.d_bad);
> -	ptvar_free(ve.rvstate);
>  	return moveon;
>  
>  out_pool:
> @@ -498,8 +490,7 @@ _("Could not create media verifier."));
>  	bitmap_free(&ve.r_bad);
>  out_dbad:
>  	bitmap_free(&ve.d_bad);
> -out_ve:
> -	ptvar_free(ve.rvstate);
> +out:
>  	return moveon;
>  }
>  
> diff --git a/scrub/read_verify.c b/scrub/read_verify.c
> index 75cb53ca..b5774736 100644
> --- a/scrub/read_verify.c
> +++ b/scrub/read_verify.c
> @@ -7,6 +7,7 @@
>  #include <stdint.h>
>  #include <stdlib.h>
>  #include <sys/statvfs.h>
> +#include "ptvar.h"
>  #include "workqueue.h"
>  #include "path.h"
>  #include "xfs_scrub.h"
> @@ -36,22 +37,40 @@
>  /* Tolerate 64k holes in adjacent read verify requests. */
>  #define RVP_IO_BATCH_LOCALITY	(65536)
>  
> +struct read_verify {
> +	void			*io_end_arg;
> +	struct disk		*io_disk;
> +	uint64_t		io_start;	/* bytes */
> +	uint64_t		io_length;	/* bytes */
> +};
> +
>  struct read_verify_pool {
>  	struct workqueue	wq;		/* thread pool */
>  	struct scrub_ctx	*ctx;		/* scrub context */
>  	void			*readbuf;	/* read buffer */
>  	struct ptcounter	*verified_bytes;
> +	struct ptvar		*rvstate;	/* combines read requests */
>  	read_verify_ioerr_fn_t	ioerr_fn;	/* io error callback */
>  	size_t			miniosz;	/* minimum io size, bytes */
>  };
>  
> -/* Create a thread pool to run read verifiers. */
> +/*
> + * Create a thread pool to run read verifiers.
> + *
> + * @miniosz is the minimum size of an IO to expect (in bytes).
> + * @ioerr_fn will be called when IO errors occur.
> + * @nproc is the maximum number of verify requests that may be sent to a disk
> + * at any given time.
> + * @submitter_threads is the number of threads that may be sending verify
> + * requests at any given time.
> + */
>  struct read_verify_pool *
>  read_verify_pool_init(
>  	struct scrub_ctx		*ctx,
>  	size_t				miniosz,
>  	read_verify_ioerr_fn_t		ioerr_fn,
> -	unsigned int			nproc)
> +	unsigned int			nproc,
> +	unsigned int			submitter_threads)
>  {
>  	struct read_verify_pool		*rvp;
>  	bool				ret;
> @@ -71,14 +90,20 @@ read_verify_pool_init(
>  	rvp->miniosz = miniosz;
>  	rvp->ctx = ctx;
>  	rvp->ioerr_fn = ioerr_fn;
> +	rvp->rvstate = ptvar_init(submitter_threads,
> +			sizeof(struct read_verify));
> +	if (rvp->rvstate == NULL)
> +		goto out_counter;
>  	/* Run in the main thread if we only want one thread. */
>  	if (nproc == 1)
>  		nproc = 0;
>  	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp, nproc);
>  	if (ret)
> -		goto out_counter;
> +		goto out_rvstate;
>  	return rvp;
>  
> +out_rvstate:
> +	ptvar_free(rvp->rvstate);
>  out_counter:
>  	ptcounter_free(rvp->verified_bytes);
>  out_buf:
> @@ -101,6 +126,7 @@ void
>  read_verify_pool_destroy(
>  	struct read_verify_pool		*rvp)
>  {
> +	ptvar_free(rvp->rvstate);
>  	ptcounter_free(rvp->verified_bytes);
>  	free(rvp->readbuf);
>  	free(rvp);
> @@ -186,16 +212,17 @@ _("Could not queue read-verify work."));
>  bool
>  read_verify_schedule_io(
>  	struct read_verify_pool		*rvp,
> -	struct read_verify		*rv,
>  	struct disk			*disk,
>  	uint64_t			start,
>  	uint64_t			length,
>  	void				*end_arg)
>  {
> +	struct read_verify		*rv;
>  	uint64_t			req_end;
>  	uint64_t			rv_end;
>  
>  	assert(rvp->readbuf);
> +	rv = ptvar_get(rvp->rvstate);
>  	req_end = start + length;
>  	rv_end = rv->io_start + rv->io_length;
>  
> @@ -229,12 +256,13 @@ read_verify_schedule_io(
>  /* Force any stashed IOs into the verifier. */
>  bool
>  read_verify_force_io(
> -	struct read_verify_pool		*rvp,
> -	struct read_verify		*rv)
> +	struct read_verify_pool		*rvp)
>  {
> +	struct read_verify		*rv;
>  	bool				moveon;
>  
>  	assert(rvp->readbuf);
> +	rv = ptvar_get(rvp->rvstate);
>  	if (rv->io_length == 0)
>  		return true;
>  
> diff --git a/scrub/read_verify.h b/scrub/read_verify.h
> index 38f1cd1a..1e7fd83f 100644
> --- a/scrub/read_verify.h
> +++ b/scrub/read_verify.h
> @@ -16,21 +16,13 @@ typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
>  
>  struct read_verify_pool *read_verify_pool_init(struct scrub_ctx *ctx,
>  		size_t miniosz, read_verify_ioerr_fn_t ioerr_fn,
> -		unsigned int nproc);
> +		unsigned int nproc, unsigned int submitter_threads);
>  void read_verify_pool_flush(struct read_verify_pool *rvp);
>  void read_verify_pool_destroy(struct read_verify_pool *rvp);
>  
> -struct read_verify {
> -	void			*io_end_arg;
> -	struct disk		*io_disk;
> -	uint64_t		io_start;	/* bytes */
> -	uint64_t		io_length;	/* bytes */
> -};
> -
> -bool read_verify_schedule_io(struct read_verify_pool *rvp,
> -		struct read_verify *rv, struct disk *disk, uint64_t start,
> -		uint64_t length, void *end_arg);
> -bool read_verify_force_io(struct read_verify_pool *rvp, struct read_verify *rv);
> +bool read_verify_schedule_io(struct read_verify_pool *rvp, struct disk *disk,
> +		uint64_t start, uint64_t length, void *end_arg);
> +bool read_verify_force_io(struct read_verify_pool *rvp);
>  uint64_t read_verify_bytes(struct read_verify_pool *rvp);
>  
>  #endif /* XFS_SCRUB_READ_VERIFY_H_ */
> 

[PATCH 15/36] xfs_scrub: one read/verify pool per disk

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Simplify the read/verify pool code further by creating one pool per
disk.  This enables us to tailor the concurrency levels of each disk to
that specific disk so that if we have a mixed hdd/ssd environment we
don't flood the hdd with a lot of requests.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase6.c      |  110 ++++++++++++++++++++++++++++++++++++---------------
 scrub/read_verify.c |   29 ++++++-------
 scrub/read_verify.h |   10 +++--
 3 files changed, 98 insertions(+), 51 deletions(-)


diff --git a/scrub/phase6.c b/scrub/phase6.c
index fe121769..ccb795ab 100644
--- a/scrub/phase6.c
+++ b/scrub/phase6.c
@@ -33,18 +33,29 @@
  * and report the paths of the now corrupt files.
  */
 
+/* Verify disk blocks with GETFSMAP */
+
+struct xfs_verify_extent {
+	struct read_verify_pool	*rvp_data;
+	struct read_verify_pool	*rvp_log;
+	struct read_verify_pool	*rvp_realtime;
+	struct bitmap		*d_bad;		/* bytes */
+	struct bitmap		*r_bad;		/* bytes */
+};
+
 /* Find the fd for a given device identifier. */
-static struct disk *
-xfs_dev_to_disk(
-	struct scrub_ctx	*ctx,
-	dev_t			dev)
+static struct read_verify_pool *
+xfs_dev_to_pool(
+	struct scrub_ctx		*ctx,
+	struct xfs_verify_extent	*ve,
+	dev_t				dev)
 {
 	if (dev == ctx->fsinfo.fs_datadev)
-		return ctx->datadev;
+		return ve->rvp_data;
 	else if (dev == ctx->fsinfo.fs_logdev)
-		return ctx->logdev;
+		return ve->rvp_log;
 	else if (dev == ctx->fsinfo.fs_rtdev)
-		return ctx->rtdev;
+		return ve->rvp_realtime;
 	abort();
 }
 
@@ -285,14 +296,6 @@ xfs_report_verify_errors(
 	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
 }
 
-/* Verify disk blocks with GETFSMAP */
-
-struct xfs_verify_extent {
-	struct read_verify_pool	*readverify;
-	struct bitmap		*d_bad;		/* bytes */
-	struct bitmap		*r_bad;		/* bytes */
-};
-
 /* Report an IO error resulting from read-verify based off getfsmap. */
 static bool
 xfs_check_rmap_error_report(
@@ -393,7 +396,9 @@ xfs_check_rmap(
 	void				*arg)
 {
 	struct xfs_verify_extent	*ve = arg;
-	struct disk			*disk;
+	struct read_verify_pool		*rvp;
+
+	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
 
 	dbg_printf("rmap dev %d:%d phys %"PRIu64" owner %"PRId64
 			" offset %"PRIu64" len %"PRIu64" flags 0x%x\n",
@@ -420,19 +425,32 @@ xfs_check_rmap(
 	/* XXX: Filter out directory data blocks. */
 
 	/* Schedule the read verify command for (eventual) running. */
-	disk = xfs_dev_to_disk(ctx, map->fmr_device);
-
-	read_verify_schedule_io(ve->readverify, disk, map->fmr_physical,
-			map->fmr_length, ve);
+	read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, ve);
 
 out:
 	/* Is this the last extent?  Fire off the read. */
 	if (map->fmr_flags & FMR_OF_LAST)
-		read_verify_force_io(ve->readverify);
+		read_verify_force_io(rvp);
 
 	return true;
 }
 
+/* Wait for read/verify actions to finish, then return # bytes checked. */
+static uint64_t
+clean_pool(
+	struct read_verify_pool	*rvp)
+{
+	uint64_t		ret;
+
+	if (!rvp)
+		return 0;
+
+	read_verify_pool_flush(rvp);
+	ret += read_verify_bytes(rvp);
+	read_verify_pool_destroy(rvp);
+	return ret;
+}
+
 /*
  * Read verify all the file data blocks in a filesystem.  Since XFS doesn't
  * do data checksums, we trust that the underlying storage will pass back
@@ -445,7 +463,7 @@ bool
 xfs_scan_blocks(
 	struct scrub_ctx		*ctx)
 {
-	struct xfs_verify_extent	ve;
+	struct xfs_verify_extent	ve = { NULL };
 	bool				moveon;
 
 	moveon = bitmap_init(&ve.d_bad);
@@ -460,21 +478,43 @@ xfs_scan_blocks(
 		goto out_dbad;
 	}
 
-	ve.readverify = read_verify_pool_init(ctx, ctx->geo.blocksize,
-			xfs_check_rmap_ioerr, disk_heads(ctx->datadev),
+	ve.rvp_data = read_verify_pool_init(ctx, ctx->datadev,
+			ctx->geo.blocksize, xfs_check_rmap_ioerr,
 			scrub_nproc(ctx));
-	if (!ve.readverify) {
+	if (!ve.rvp_data) {
 		moveon = false;
 		str_info(ctx, ctx->mntpoint,
-_("Could not create media verifier."));
+_("Could not create data device media verifier."));
 		goto out_rbad;
 	}
+	if (ctx->logdev) {
+		ve.rvp_log = read_verify_pool_init(ctx, ctx->logdev,
+				ctx->geo.blocksize, xfs_check_rmap_ioerr,
+				scrub_nproc(ctx));
+		if (!ve.rvp_log) {
+			moveon = false;
+			str_info(ctx, ctx->mntpoint,
+	_("Could not create log device media verifier."));
+			goto out_datapool;
+		}
+	}
+	if (ctx->rtdev) {
+		ve.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev,
+				ctx->geo.blocksize, xfs_check_rmap_ioerr,
+				scrub_nproc(ctx));
+		if (!ve.rvp_realtime) {
+			moveon = false;
+			str_info(ctx, ctx->mntpoint,
+	_("Could not create realtime device media verifier."));
+			goto out_logpool;
+		}
+	}
 	moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &ve);
 	if (!moveon)
-		goto out_pool;
-	read_verify_pool_flush(ve.readverify);
-	ctx->bytes_checked += read_verify_bytes(ve.readverify);
-	read_verify_pool_destroy(ve.readverify);
+		goto out_rtpool;
+	ctx->bytes_checked += clean_pool(ve.rvp_data);
+	ctx->bytes_checked += clean_pool(ve.rvp_log);
+	ctx->bytes_checked += clean_pool(ve.rvp_realtime);
 
 	/* Scan the whole dir tree to see what matches the bad extents. */
 	if (!bitmap_empty(ve.d_bad) || !bitmap_empty(ve.r_bad))
@@ -484,8 +524,14 @@ _("Could not create media verifier."));
 	bitmap_free(&ve.d_bad);
 	return moveon;
 
-out_pool:
-	read_verify_pool_destroy(ve.readverify);
+out_rtpool:
+	if (ve.rvp_realtime)
+		read_verify_pool_destroy(ve.rvp_realtime);
+out_logpool:
+	if (ve.rvp_log)
+		read_verify_pool_destroy(ve.rvp_log);
+out_datapool:
+	read_verify_pool_destroy(ve.rvp_data);
 out_rbad:
 	bitmap_free(&ve.r_bad);
 out_dbad:
diff --git a/scrub/read_verify.c b/scrub/read_verify.c
index b5774736..4a9b91f2 100644
--- a/scrub/read_verify.c
+++ b/scrub/read_verify.c
@@ -50,6 +50,7 @@ struct read_verify_pool {
 	void			*readbuf;	/* read buffer */
 	struct ptcounter	*verified_bytes;
 	struct ptvar		*rvstate;	/* combines read requests */
+	struct disk		*disk;		/* which disk? */
 	read_verify_ioerr_fn_t	ioerr_fn;	/* io error callback */
 	size_t			miniosz;	/* minimum io size, bytes */
 };
@@ -57,19 +58,18 @@ struct read_verify_pool {
 /*
  * Create a thread pool to run read verifiers.
  *
+ * @disk is the disk we want to verify.
  * @miniosz is the minimum size of an IO to expect (in bytes).
  * @ioerr_fn will be called when IO errors occur.
- * @nproc is the maximum number of verify requests that may be sent to a disk
- * at any given time.
  * @submitter_threads is the number of threads that may be sending verify
  * requests at any given time.
  */
 struct read_verify_pool *
 read_verify_pool_init(
 	struct scrub_ctx		*ctx,
+	struct disk			*disk,
 	size_t				miniosz,
 	read_verify_ioerr_fn_t		ioerr_fn,
-	unsigned int			nproc,
 	unsigned int			submitter_threads)
 {
 	struct read_verify_pool		*rvp;
@@ -89,6 +89,7 @@ read_verify_pool_init(
 		goto out_buf;
 	rvp->miniosz = miniosz;
 	rvp->ctx = ctx;
+	rvp->disk = disk;
 	rvp->ioerr_fn = ioerr_fn;
 	rvp->rvstate = ptvar_init(submitter_threads,
 			sizeof(struct read_verify));
@@ -97,7 +98,8 @@ read_verify_pool_init(
 	/* Run in the main thread if we only want one thread. */
 	if (nproc == 1)
 		nproc = 0;
-	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp, nproc);
+	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp,
+			disk_heads(disk));
 	if (ret)
 		goto out_rvstate;
 	return rvp;
@@ -150,17 +152,16 @@ read_verify(
 	rvp = (struct read_verify_pool *)wq->wq_ctx;
 	while (rv->io_length > 0) {
 		len = min(rv->io_length, RVP_IO_MAX_SIZE);
-		dbg_printf("diskverify %d %"PRIu64" %zu\n", rv->io_disk->d_fd,
-				rv->io_start, len);
-		sz = disk_read_verify(rv->io_disk, rvp->readbuf,
+		dbg_printf("diskverify %d %"PRIu64" %zu\n", rvp->disk->d_fd,
 				rv->io_start, len);
+		sz = disk_read_verify(rvp->disk, rvp->readbuf, rv->io_start,
+				len);
 		if (sz < 0) {
 			dbg_printf("IOERR %d %"PRIu64" %zu\n",
-					rv->io_disk->d_fd,
-					rv->io_start, len);
+					rvp->disk->d_fd, rv->io_start, len);
 			/* IO error, so try the next logical block. */
 			len = rvp->miniosz;
-			rvp->ioerr_fn(rvp->ctx, rv->io_disk, rv->io_start, len,
+			rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start, len,
 					errno, rv->io_end_arg);
 		}
 
@@ -184,11 +185,11 @@ read_verify_queue(
 	bool				ret;
 
 	dbg_printf("verify fd %d start %"PRIu64" len %"PRIu64"\n",
-			rv->io_disk->d_fd, rv->io_start, rv->io_length);
+			rvp->disk->d_fd, rv->io_start, rv->io_length);
 
 	tmp = malloc(sizeof(struct read_verify));
 	if (!tmp) {
-		rvp->ioerr_fn(rvp->ctx, rv->io_disk, rv->io_start,
+		rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start,
 				rv->io_length, errno, rv->io_end_arg);
 		return true;
 	}
@@ -212,7 +213,6 @@ _("Could not queue read-verify work."));
 bool
 read_verify_schedule_io(
 	struct read_verify_pool		*rvp,
-	struct disk			*disk,
 	uint64_t			start,
 	uint64_t			length,
 	void				*end_arg)
@@ -231,7 +231,7 @@ read_verify_schedule_io(
 	 * reporting is the same, and the two extents are close,
 	 * we can combine them.
 	 */
-	if (rv->io_length > 0 && disk == rv->io_disk &&
+	if (rv->io_length > 0 &&
 	    end_arg == rv->io_end_arg &&
 	    ((start >= rv->io_start && start <= rv_end + RVP_IO_BATCH_LOCALITY) ||
 	     (rv->io_start >= start &&
@@ -244,7 +244,6 @@ read_verify_schedule_io(
 			return read_verify_queue(rvp, rv);
 
 		/* Stash the new IO. */
-		rv->io_disk = disk;
 		rv->io_start = start;
 		rv->io_length = length;
 		rv->io_end_arg = end_arg;
diff --git a/scrub/read_verify.h b/scrub/read_verify.h
index 1e7fd83f..5fabe5e0 100644
--- a/scrub/read_verify.h
+++ b/scrub/read_verify.h
@@ -8,6 +8,7 @@
 
 struct scrub_ctx;
 struct read_verify_pool;
+struct disk;
 
 /* Function called when an IO error happens. */
 typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
@@ -15,13 +16,14 @@ typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
 		int error, void *arg);
 
 struct read_verify_pool *read_verify_pool_init(struct scrub_ctx *ctx,
-		size_t miniosz, read_verify_ioerr_fn_t ioerr_fn,
-		unsigned int nproc, unsigned int submitter_threads);
+		struct disk *disk, size_t miniosz,
+		read_verify_ioerr_fn_t ioerr_fn,
+		unsigned int submitter_threads);
 void read_verify_pool_flush(struct read_verify_pool *rvp);
 void read_verify_pool_destroy(struct read_verify_pool *rvp);
 
-bool read_verify_schedule_io(struct read_verify_pool *rvp, struct disk *disk,
-		uint64_t start, uint64_t length, void *end_arg);
+bool read_verify_schedule_io(struct read_verify_pool *rvp, uint64_t start,
+		uint64_t length, void *end_arg);
 bool read_verify_force_io(struct read_verify_pool *rvp);
 uint64_t read_verify_bytes(struct read_verify_pool *rvp);
 

Re: [PATCH 15/36] xfs_scrub: one read/verify pool per disk

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Simplify the read/verify pool code further by creating one pool per
> disk.  This enables us to tailor the concurrency levels of each disk to
> that specific disk so that if we have a mixed hdd/ssd environment we
> don't flood the hdd with a lot of requests.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Spotted one issue below

> ---
>  scrub/phase6.c      |  110 ++++++++++++++++++++++++++++++++++++---------------
>  scrub/read_verify.c |   29 ++++++-------
>  scrub/read_verify.h |   10 +++--
>  3 files changed, 98 insertions(+), 51 deletions(-)
> 
> 
> diff --git a/scrub/phase6.c b/scrub/phase6.c
> index fe121769..ccb795ab 100644
> --- a/scrub/phase6.c
> +++ b/scrub/phase6.c
> @@ -33,18 +33,29 @@
>   * and report the paths of the now corrupt files.
>   */
>  
> +/* Verify disk blocks with GETFSMAP */
> +
> +struct xfs_verify_extent {
> +	struct read_verify_pool	*rvp_data;
> +	struct read_verify_pool	*rvp_log;
> +	struct read_verify_pool	*rvp_realtime;
> +	struct bitmap		*d_bad;		/* bytes */
> +	struct bitmap		*r_bad;		/* bytes */
> +};
> +
>  /* Find the fd for a given device identifier. */
> -static struct disk *
> -xfs_dev_to_disk(
> -	struct scrub_ctx	*ctx,
> -	dev_t			dev)
> +static struct read_verify_pool *
> +xfs_dev_to_pool(
> +	struct scrub_ctx		*ctx,
> +	struct xfs_verify_extent	*ve,
> +	dev_t				dev)
>  {
>  	if (dev == ctx->fsinfo.fs_datadev)
> -		return ctx->datadev;
> +		return ve->rvp_data;
>  	else if (dev == ctx->fsinfo.fs_logdev)
> -		return ctx->logdev;
> +		return ve->rvp_log;
>  	else if (dev == ctx->fsinfo.fs_rtdev)
> -		return ctx->rtdev;
> +		return ve->rvp_realtime;
>  	abort();
>  }
>  
> @@ -285,14 +296,6 @@ xfs_report_verify_errors(
>  	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
>  }
>  
> -/* Verify disk blocks with GETFSMAP */
> -
> -struct xfs_verify_extent {
> -	struct read_verify_pool	*readverify;
> -	struct bitmap		*d_bad;		/* bytes */
> -	struct bitmap		*r_bad;		/* bytes */
> -};
> -
>  /* Report an IO error resulting from read-verify based off getfsmap. */
>  static bool
>  xfs_check_rmap_error_report(
> @@ -393,7 +396,9 @@ xfs_check_rmap(
>  	void				*arg)
>  {
>  	struct xfs_verify_extent	*ve = arg;
> -	struct disk			*disk;
> +	struct read_verify_pool		*rvp;
> +
> +	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
>  
>  	dbg_printf("rmap dev %d:%d phys %"PRIu64" owner %"PRId64
>  			" offset %"PRIu64" len %"PRIu64" flags 0x%x\n",
> @@ -420,19 +425,32 @@ xfs_check_rmap(
>  	/* XXX: Filter out directory data blocks. */
>  
>  	/* Schedule the read verify command for (eventual) running. */
> -	disk = xfs_dev_to_disk(ctx, map->fmr_device);
> -
> -	read_verify_schedule_io(ve->readverify, disk, map->fmr_physical,
> -			map->fmr_length, ve);
> +	read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, ve);
>  
>  out:
>  	/* Is this the last extent?  Fire off the read. */
>  	if (map->fmr_flags & FMR_OF_LAST)
> -		read_verify_force_io(ve->readverify);
> +		read_verify_force_io(rvp);
>  
>  	return true;
>  }
>  
> +/* Wait for read/verify actions to finish, then return # bytes checked. */
> +static uint64_t
> +clean_pool(
> +	struct read_verify_pool	*rvp)
> +{
> +	uint64_t		ret;
> +
> +	if (!rvp)
> +		return 0;
> +
> +	read_verify_pool_flush(rvp);
> +	ret += read_verify_bytes(rvp);

ret is uninitialized, += adds to uninitialized value ...

> +	read_verify_pool_destroy(rvp);
> +	return ret;
> +}
> +
>  /*
>   * Read verify all the file data blocks in a filesystem.  Since XFS doesn't
>   * do data checksums, we trust that the underlying storage will pass back
> @@ -445,7 +463,7 @@ bool
>  xfs_scan_blocks(
>  	struct scrub_ctx		*ctx)
>  {
> -	struct xfs_verify_extent	ve;
> +	struct xfs_verify_extent	ve = { NULL };
>  	bool				moveon;
>  
>  	moveon = bitmap_init(&ve.d_bad);
> @@ -460,21 +478,43 @@ xfs_scan_blocks(
>  		goto out_dbad;
>  	}
>  
> -	ve.readverify = read_verify_pool_init(ctx, ctx->geo.blocksize,
> -			xfs_check_rmap_ioerr, disk_heads(ctx->datadev),
> +	ve.rvp_data = read_verify_pool_init(ctx, ctx->datadev,
> +			ctx->geo.blocksize, xfs_check_rmap_ioerr,
>  			scrub_nproc(ctx));
> -	if (!ve.readverify) {
> +	if (!ve.rvp_data) {
>  		moveon = false;
>  		str_info(ctx, ctx->mntpoint,
> -_("Could not create media verifier."));
> +_("Could not create data device media verifier."));
>  		goto out_rbad;
>  	}
> +	if (ctx->logdev) {
> +		ve.rvp_log = read_verify_pool_init(ctx, ctx->logdev,
> +				ctx->geo.blocksize, xfs_check_rmap_ioerr,
> +				scrub_nproc(ctx));
> +		if (!ve.rvp_log) {
> +			moveon = false;
> +			str_info(ctx, ctx->mntpoint,
> +	_("Could not create log device media verifier."));
> +			goto out_datapool;
> +		}
> +	}
> +	if (ctx->rtdev) {
> +		ve.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev,
> +				ctx->geo.blocksize, xfs_check_rmap_ioerr,
> +				scrub_nproc(ctx));
> +		if (!ve.rvp_realtime) {
> +			moveon = false;
> +			str_info(ctx, ctx->mntpoint,
> +	_("Could not create realtime device media verifier."));
> +			goto out_logpool;
> +		}
> +	}
>  	moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &ve);
>  	if (!moveon)
> -		goto out_pool;
> -	read_verify_pool_flush(ve.readverify);
> -	ctx->bytes_checked += read_verify_bytes(ve.readverify);
> -	read_verify_pool_destroy(ve.readverify);
> +		goto out_rtpool;
> +	ctx->bytes_checked += clean_pool(ve.rvp_data);
> +	ctx->bytes_checked += clean_pool(ve.rvp_log);
> +	ctx->bytes_checked += clean_pool(ve.rvp_realtime);
>  
>  	/* Scan the whole dir tree to see what matches the bad extents. */
>  	if (!bitmap_empty(ve.d_bad) || !bitmap_empty(ve.r_bad))
> @@ -484,8 +524,14 @@ _("Could not create media verifier."));
>  	bitmap_free(&ve.d_bad);
>  	return moveon;
>  
> -out_pool:
> -	read_verify_pool_destroy(ve.readverify);
> +out_rtpool:
> +	if (ve.rvp_realtime)
> +		read_verify_pool_destroy(ve.rvp_realtime);
> +out_logpool:
> +	if (ve.rvp_log)
> +		read_verify_pool_destroy(ve.rvp_log);
> +out_datapool:
> +	read_verify_pool_destroy(ve.rvp_data);
>  out_rbad:
>  	bitmap_free(&ve.r_bad);
>  out_dbad:
> diff --git a/scrub/read_verify.c b/scrub/read_verify.c
> index b5774736..4a9b91f2 100644
> --- a/scrub/read_verify.c
> +++ b/scrub/read_verify.c
> @@ -50,6 +50,7 @@ struct read_verify_pool {
>  	void			*readbuf;	/* read buffer */
>  	struct ptcounter	*verified_bytes;
>  	struct ptvar		*rvstate;	/* combines read requests */
> +	struct disk		*disk;		/* which disk? */
>  	read_verify_ioerr_fn_t	ioerr_fn;	/* io error callback */
>  	size_t			miniosz;	/* minimum io size, bytes */
>  };
> @@ -57,19 +58,18 @@ struct read_verify_pool {
>  /*
>   * Create a thread pool to run read verifiers.
>   *
> + * @disk is the disk we want to verify.
>   * @miniosz is the minimum size of an IO to expect (in bytes).
>   * @ioerr_fn will be called when IO errors occur.
> - * @nproc is the maximum number of verify requests that may be sent to a disk
> - * at any given time.
>   * @submitter_threads is the number of threads that may be sending verify
>   * requests at any given time.
>   */
>  struct read_verify_pool *
>  read_verify_pool_init(
>  	struct scrub_ctx		*ctx,
> +	struct disk			*disk,
>  	size_t				miniosz,
>  	read_verify_ioerr_fn_t		ioerr_fn,
> -	unsigned int			nproc,
>  	unsigned int			submitter_threads)
>  {
>  	struct read_verify_pool		*rvp;
> @@ -89,6 +89,7 @@ read_verify_pool_init(
>  		goto out_buf;
>  	rvp->miniosz = miniosz;
>  	rvp->ctx = ctx;
> +	rvp->disk = disk;
>  	rvp->ioerr_fn = ioerr_fn;
>  	rvp->rvstate = ptvar_init(submitter_threads,
>  			sizeof(struct read_verify));
> @@ -97,7 +98,8 @@ read_verify_pool_init(
>  	/* Run in the main thread if we only want one thread. */
>  	if (nproc == 1)
>  		nproc = 0;
> -	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp, nproc);
> +	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp,
> +			disk_heads(disk));
>  	if (ret)
>  		goto out_rvstate;
>  	return rvp;
> @@ -150,17 +152,16 @@ read_verify(
>  	rvp = (struct read_verify_pool *)wq->wq_ctx;
>  	while (rv->io_length > 0) {
>  		len = min(rv->io_length, RVP_IO_MAX_SIZE);
> -		dbg_printf("diskverify %d %"PRIu64" %zu\n", rv->io_disk->d_fd,
> -				rv->io_start, len);
> -		sz = disk_read_verify(rv->io_disk, rvp->readbuf,
> +		dbg_printf("diskverify %d %"PRIu64" %zu\n", rvp->disk->d_fd,
>  				rv->io_start, len);
> +		sz = disk_read_verify(rvp->disk, rvp->readbuf, rv->io_start,
> +				len);
>  		if (sz < 0) {
>  			dbg_printf("IOERR %d %"PRIu64" %zu\n",
> -					rv->io_disk->d_fd,
> -					rv->io_start, len);
> +					rvp->disk->d_fd, rv->io_start, len);
>  			/* IO error, so try the next logical block. */
>  			len = rvp->miniosz;
> -			rvp->ioerr_fn(rvp->ctx, rv->io_disk, rv->io_start, len,
> +			rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start, len,
>  					errno, rv->io_end_arg);
>  		}
>  
> @@ -184,11 +185,11 @@ read_verify_queue(
>  	bool				ret;
>  
>  	dbg_printf("verify fd %d start %"PRIu64" len %"PRIu64"\n",
> -			rv->io_disk->d_fd, rv->io_start, rv->io_length);
> +			rvp->disk->d_fd, rv->io_start, rv->io_length);
>  
>  	tmp = malloc(sizeof(struct read_verify));
>  	if (!tmp) {
> -		rvp->ioerr_fn(rvp->ctx, rv->io_disk, rv->io_start,
> +		rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start,
>  				rv->io_length, errno, rv->io_end_arg);
>  		return true;
>  	}
> @@ -212,7 +213,6 @@ _("Could not queue read-verify work."));
>  bool
>  read_verify_schedule_io(
>  	struct read_verify_pool		*rvp,
> -	struct disk			*disk,
>  	uint64_t			start,
>  	uint64_t			length,
>  	void				*end_arg)
> @@ -231,7 +231,7 @@ read_verify_schedule_io(
>  	 * reporting is the same, and the two extents are close,
>  	 * we can combine them.
>  	 */
> -	if (rv->io_length > 0 && disk == rv->io_disk &&
> +	if (rv->io_length > 0 &&
>  	    end_arg == rv->io_end_arg &&
>  	    ((start >= rv->io_start && start <= rv_end + RVP_IO_BATCH_LOCALITY) ||
>  	     (rv->io_start >= start &&
> @@ -244,7 +244,6 @@ read_verify_schedule_io(
>  			return read_verify_queue(rvp, rv);
>  
>  		/* Stash the new IO. */
> -		rv->io_disk = disk;
>  		rv->io_start = start;
>  		rv->io_length = length;
>  		rv->io_end_arg = end_arg;
> diff --git a/scrub/read_verify.h b/scrub/read_verify.h
> index 1e7fd83f..5fabe5e0 100644
> --- a/scrub/read_verify.h
> +++ b/scrub/read_verify.h
> @@ -8,6 +8,7 @@
>  
>  struct scrub_ctx;
>  struct read_verify_pool;
> +struct disk;
>  
>  /* Function called when an IO error happens. */
>  typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
> @@ -15,13 +16,14 @@ typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
>  		int error, void *arg);
>  
>  struct read_verify_pool *read_verify_pool_init(struct scrub_ctx *ctx,
> -		size_t miniosz, read_verify_ioerr_fn_t ioerr_fn,
> -		unsigned int nproc, unsigned int submitter_threads);
> +		struct disk *disk, size_t miniosz,
> +		read_verify_ioerr_fn_t ioerr_fn,
> +		unsigned int submitter_threads);
>  void read_verify_pool_flush(struct read_verify_pool *rvp);
>  void read_verify_pool_destroy(struct read_verify_pool *rvp);
>  
> -bool read_verify_schedule_io(struct read_verify_pool *rvp, struct disk *disk,
> -		uint64_t start, uint64_t length, void *end_arg);
> +bool read_verify_schedule_io(struct read_verify_pool *rvp, uint64_t start,
> +		uint64_t length, void *end_arg);
>  bool read_verify_force_io(struct read_verify_pool *rvp);
>  uint64_t read_verify_bytes(struct read_verify_pool *rvp);
>  
> 

[PATCH v2 15/36] xfs_scrub: one read/verify pool per disk

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Simplify the read/verify pool code further by creating one pool per
disk.  This enables us to tailor the concurrency levels of each disk to
that specific disk so that if we have a mixed hdd/ssd environment we
don't flood the hdd with a lot of requests.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
v2: fix uninitialized variable error pointed out by Eric Sandeen
---
 scrub/phase6.c      |  110 ++++++++++++++++++++++++++++++++++++---------------
 scrub/read_verify.c |   29 ++++++-------
 scrub/read_verify.h |   10 +++--
 3 files changed, 98 insertions(+), 51 deletions(-)

diff --git a/scrub/phase6.c b/scrub/phase6.c
index fe121769..5e2b9ee3 100644
--- a/scrub/phase6.c
+++ b/scrub/phase6.c
@@ -33,18 +33,29 @@
  * and report the paths of the now corrupt files.
  */
 
+/* Verify disk blocks with GETFSMAP */
+
+struct xfs_verify_extent {
+	struct read_verify_pool	*rvp_data;
+	struct read_verify_pool	*rvp_log;
+	struct read_verify_pool	*rvp_realtime;
+	struct bitmap		*d_bad;		/* bytes */
+	struct bitmap		*r_bad;		/* bytes */
+};
+
 /* Find the fd for a given device identifier. */
-static struct disk *
-xfs_dev_to_disk(
-	struct scrub_ctx	*ctx,
-	dev_t			dev)
+static struct read_verify_pool *
+xfs_dev_to_pool(
+	struct scrub_ctx		*ctx,
+	struct xfs_verify_extent	*ve,
+	dev_t				dev)
 {
 	if (dev == ctx->fsinfo.fs_datadev)
-		return ctx->datadev;
+		return ve->rvp_data;
 	else if (dev == ctx->fsinfo.fs_logdev)
-		return ctx->logdev;
+		return ve->rvp_log;
 	else if (dev == ctx->fsinfo.fs_rtdev)
-		return ctx->rtdev;
+		return ve->rvp_realtime;
 	abort();
 }
 
@@ -285,14 +296,6 @@ xfs_report_verify_errors(
 	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
 }
 
-/* Verify disk blocks with GETFSMAP */
-
-struct xfs_verify_extent {
-	struct read_verify_pool	*readverify;
-	struct bitmap		*d_bad;		/* bytes */
-	struct bitmap		*r_bad;		/* bytes */
-};
-
 /* Report an IO error resulting from read-verify based off getfsmap. */
 static bool
 xfs_check_rmap_error_report(
@@ -393,7 +396,9 @@ xfs_check_rmap(
 	void				*arg)
 {
 	struct xfs_verify_extent	*ve = arg;
-	struct disk			*disk;
+	struct read_verify_pool		*rvp;
+
+	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
 
 	dbg_printf("rmap dev %d:%d phys %"PRIu64" owner %"PRId64
 			" offset %"PRIu64" len %"PRIu64" flags 0x%x\n",
@@ -420,19 +425,32 @@ xfs_check_rmap(
 	/* XXX: Filter out directory data blocks. */
 
 	/* Schedule the read verify command for (eventual) running. */
-	disk = xfs_dev_to_disk(ctx, map->fmr_device);
-
-	read_verify_schedule_io(ve->readverify, disk, map->fmr_physical,
-			map->fmr_length, ve);
+	read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, ve);
 
 out:
 	/* Is this the last extent?  Fire off the read. */
 	if (map->fmr_flags & FMR_OF_LAST)
-		read_verify_force_io(ve->readverify);
+		read_verify_force_io(rvp);
 
 	return true;
 }
 
+/* Wait for read/verify actions to finish, then return # bytes checked. */
+static uint64_t
+clean_pool(
+	struct read_verify_pool	*rvp)
+{
+	uint64_t		ret;
+
+	if (!rvp)
+		return 0;
+
+	read_verify_pool_flush(rvp);
+	ret = read_verify_bytes(rvp);
+	read_verify_pool_destroy(rvp);
+	return ret;
+}
+
 /*
  * Read verify all the file data blocks in a filesystem.  Since XFS doesn't
  * do data checksums, we trust that the underlying storage will pass back
@@ -445,7 +463,7 @@ bool
 xfs_scan_blocks(
 	struct scrub_ctx		*ctx)
 {
-	struct xfs_verify_extent	ve;
+	struct xfs_verify_extent	ve = { NULL };
 	bool				moveon;
 
 	moveon = bitmap_init(&ve.d_bad);
@@ -460,21 +478,43 @@ xfs_scan_blocks(
 		goto out_dbad;
 	}
 
-	ve.readverify = read_verify_pool_init(ctx, ctx->geo.blocksize,
-			xfs_check_rmap_ioerr, disk_heads(ctx->datadev),
+	ve.rvp_data = read_verify_pool_init(ctx, ctx->datadev,
+			ctx->geo.blocksize, xfs_check_rmap_ioerr,
 			scrub_nproc(ctx));
-	if (!ve.readverify) {
+	if (!ve.rvp_data) {
 		moveon = false;
 		str_info(ctx, ctx->mntpoint,
-_("Could not create media verifier."));
+_("Could not create data device media verifier."));
 		goto out_rbad;
 	}
+	if (ctx->logdev) {
+		ve.rvp_log = read_verify_pool_init(ctx, ctx->logdev,
+				ctx->geo.blocksize, xfs_check_rmap_ioerr,
+				scrub_nproc(ctx));
+		if (!ve.rvp_log) {
+			moveon = false;
+			str_info(ctx, ctx->mntpoint,
+	_("Could not create log device media verifier."));
+			goto out_datapool;
+		}
+	}
+	if (ctx->rtdev) {
+		ve.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev,
+				ctx->geo.blocksize, xfs_check_rmap_ioerr,
+				scrub_nproc(ctx));
+		if (!ve.rvp_realtime) {
+			moveon = false;
+			str_info(ctx, ctx->mntpoint,
+	_("Could not create realtime device media verifier."));
+			goto out_logpool;
+		}
+	}
 	moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &ve);
 	if (!moveon)
-		goto out_pool;
-	read_verify_pool_flush(ve.readverify);
-	ctx->bytes_checked += read_verify_bytes(ve.readverify);
-	read_verify_pool_destroy(ve.readverify);
+		goto out_rtpool;
+	ctx->bytes_checked += clean_pool(ve.rvp_data);
+	ctx->bytes_checked += clean_pool(ve.rvp_log);
+	ctx->bytes_checked += clean_pool(ve.rvp_realtime);
 
 	/* Scan the whole dir tree to see what matches the bad extents. */
 	if (!bitmap_empty(ve.d_bad) || !bitmap_empty(ve.r_bad))
@@ -484,8 +524,14 @@ _("Could not create media verifier."));
 	bitmap_free(&ve.d_bad);
 	return moveon;
 
-out_pool:
-	read_verify_pool_destroy(ve.readverify);
+out_rtpool:
+	if (ve.rvp_realtime)
+		read_verify_pool_destroy(ve.rvp_realtime);
+out_logpool:
+	if (ve.rvp_log)
+		read_verify_pool_destroy(ve.rvp_log);
+out_datapool:
+	read_verify_pool_destroy(ve.rvp_data);
 out_rbad:
 	bitmap_free(&ve.r_bad);
 out_dbad:
diff --git a/scrub/read_verify.c b/scrub/read_verify.c
index b5774736..4a9b91f2 100644
--- a/scrub/read_verify.c
+++ b/scrub/read_verify.c
@@ -50,6 +50,7 @@ struct read_verify_pool {
 	void			*readbuf;	/* read buffer */
 	struct ptcounter	*verified_bytes;
 	struct ptvar		*rvstate;	/* combines read requests */
+	struct disk		*disk;		/* which disk? */
 	read_verify_ioerr_fn_t	ioerr_fn;	/* io error callback */
 	size_t			miniosz;	/* minimum io size, bytes */
 };
@@ -57,19 +58,18 @@ struct read_verify_pool {
 /*
  * Create a thread pool to run read verifiers.
  *
+ * @disk is the disk we want to verify.
  * @miniosz is the minimum size of an IO to expect (in bytes).
  * @ioerr_fn will be called when IO errors occur.
- * @nproc is the maximum number of verify requests that may be sent to a disk
- * at any given time.
  * @submitter_threads is the number of threads that may be sending verify
  * requests at any given time.
  */
 struct read_verify_pool *
 read_verify_pool_init(
 	struct scrub_ctx		*ctx,
+	struct disk			*disk,
 	size_t				miniosz,
 	read_verify_ioerr_fn_t		ioerr_fn,
-	unsigned int			nproc,
 	unsigned int			submitter_threads)
 {
 	struct read_verify_pool		*rvp;
@@ -89,6 +89,7 @@ read_verify_pool_init(
 		goto out_buf;
 	rvp->miniosz = miniosz;
 	rvp->ctx = ctx;
+	rvp->disk = disk;
 	rvp->ioerr_fn = ioerr_fn;
 	rvp->rvstate = ptvar_init(submitter_threads,
 			sizeof(struct read_verify));
@@ -97,7 +98,8 @@ read_verify_pool_init(
 	/* Run in the main thread if we only want one thread. */
 	if (nproc == 1)
 		nproc = 0;
-	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp, nproc);
+	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp,
+			disk_heads(disk));
 	if (ret)
 		goto out_rvstate;
 	return rvp;
@@ -150,17 +152,16 @@ read_verify(
 	rvp = (struct read_verify_pool *)wq->wq_ctx;
 	while (rv->io_length > 0) {
 		len = min(rv->io_length, RVP_IO_MAX_SIZE);
-		dbg_printf("diskverify %d %"PRIu64" %zu\n", rv->io_disk->d_fd,
-				rv->io_start, len);
-		sz = disk_read_verify(rv->io_disk, rvp->readbuf,
+		dbg_printf("diskverify %d %"PRIu64" %zu\n", rvp->disk->d_fd,
 				rv->io_start, len);
+		sz = disk_read_verify(rvp->disk, rvp->readbuf, rv->io_start,
+				len);
 		if (sz < 0) {
 			dbg_printf("IOERR %d %"PRIu64" %zu\n",
-					rv->io_disk->d_fd,
-					rv->io_start, len);
+					rvp->disk->d_fd, rv->io_start, len);
 			/* IO error, so try the next logical block. */
 			len = rvp->miniosz;
-			rvp->ioerr_fn(rvp->ctx, rv->io_disk, rv->io_start, len,
+			rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start, len,
 					errno, rv->io_end_arg);
 		}
 
@@ -184,11 +185,11 @@ read_verify_queue(
 	bool				ret;
 
 	dbg_printf("verify fd %d start %"PRIu64" len %"PRIu64"\n",
-			rv->io_disk->d_fd, rv->io_start, rv->io_length);
+			rvp->disk->d_fd, rv->io_start, rv->io_length);
 
 	tmp = malloc(sizeof(struct read_verify));
 	if (!tmp) {
-		rvp->ioerr_fn(rvp->ctx, rv->io_disk, rv->io_start,
+		rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start,
 				rv->io_length, errno, rv->io_end_arg);
 		return true;
 	}
@@ -212,7 +213,6 @@ _("Could not queue read-verify work."));
 bool
 read_verify_schedule_io(
 	struct read_verify_pool		*rvp,
-	struct disk			*disk,
 	uint64_t			start,
 	uint64_t			length,
 	void				*end_arg)
@@ -231,7 +231,7 @@ read_verify_schedule_io(
 	 * reporting is the same, and the two extents are close,
 	 * we can combine them.
 	 */
-	if (rv->io_length > 0 && disk == rv->io_disk &&
+	if (rv->io_length > 0 &&
 	    end_arg == rv->io_end_arg &&
 	    ((start >= rv->io_start && start <= rv_end + RVP_IO_BATCH_LOCALITY) ||
 	     (rv->io_start >= start &&
@@ -244,7 +244,6 @@ read_verify_schedule_io(
 			return read_verify_queue(rvp, rv);
 
 		/* Stash the new IO. */
-		rv->io_disk = disk;
 		rv->io_start = start;
 		rv->io_length = length;
 		rv->io_end_arg = end_arg;
diff --git a/scrub/read_verify.h b/scrub/read_verify.h
index 1e7fd83f..5fabe5e0 100644
--- a/scrub/read_verify.h
+++ b/scrub/read_verify.h
@@ -8,6 +8,7 @@
 
 struct scrub_ctx;
 struct read_verify_pool;
+struct disk;
 
 /* Function called when an IO error happens. */
 typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
@@ -15,13 +16,14 @@ typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
 		int error, void *arg);
 
 struct read_verify_pool *read_verify_pool_init(struct scrub_ctx *ctx,
-		size_t miniosz, read_verify_ioerr_fn_t ioerr_fn,
-		unsigned int nproc, unsigned int submitter_threads);
+		struct disk *disk, size_t miniosz,
+		read_verify_ioerr_fn_t ioerr_fn,
+		unsigned int submitter_threads);
 void read_verify_pool_flush(struct read_verify_pool *rvp);
 void read_verify_pool_destroy(struct read_verify_pool *rvp);
 
-bool read_verify_schedule_io(struct read_verify_pool *rvp, struct disk *disk,
-		uint64_t start, uint64_t length, void *end_arg);
+bool read_verify_schedule_io(struct read_verify_pool *rvp, uint64_t start,
+		uint64_t length, void *end_arg);
 bool read_verify_force_io(struct read_verify_pool *rvp);
 uint64_t read_verify_bytes(struct read_verify_pool *rvp);
 

Re: [PATCH v2 15/36] xfs_scrub: one read/verify pool per disk

[Eric Sandeen]

On 3/20/19 3:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Simplify the read/verify pool code further by creating one pool per
> disk.  This enables us to tailor the concurrency levels of each disk to
> that specific disk so that if we have a mixed hdd/ssd environment we
> don't flood the hdd with a lot of requests.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
> v2: fix uninitialized variable error pointed out by Eric Sandeen
> ---
>  scrub/phase6.c      |  110 ++++++++++++++++++++++++++++++++++++---------------
>  scrub/read_verify.c |   29 ++++++-------
>  scrub/read_verify.h |   10 +++--
>  3 files changed, 98 insertions(+), 51 deletions(-)
> 
> diff --git a/scrub/phase6.c b/scrub/phase6.c
> index fe121769..5e2b9ee3 100644
> --- a/scrub/phase6.c
> +++ b/scrub/phase6.c
> @@ -33,18 +33,29 @@
>   * and report the paths of the now corrupt files.
>   */
>  
> +/* Verify disk blocks with GETFSMAP */
> +
> +struct xfs_verify_extent {
> +	struct read_verify_pool	*rvp_data;
> +	struct read_verify_pool	*rvp_log;
> +	struct read_verify_pool	*rvp_realtime;
> +	struct bitmap		*d_bad;		/* bytes */
> +	struct bitmap		*r_bad;		/* bytes */
> +};
> +
>  /* Find the fd for a given device identifier. */
> -static struct disk *
> -xfs_dev_to_disk(
> -	struct scrub_ctx	*ctx,
> -	dev_t			dev)
> +static struct read_verify_pool *
> +xfs_dev_to_pool(
> +	struct scrub_ctx		*ctx,
> +	struct xfs_verify_extent	*ve,
> +	dev_t				dev)
>  {
>  	if (dev == ctx->fsinfo.fs_datadev)
> -		return ctx->datadev;
> +		return ve->rvp_data;
>  	else if (dev == ctx->fsinfo.fs_logdev)
> -		return ctx->logdev;
> +		return ve->rvp_log;
>  	else if (dev == ctx->fsinfo.fs_rtdev)
> -		return ctx->rtdev;
> +		return ve->rvp_realtime;
>  	abort();
>  }
>  
> @@ -285,14 +296,6 @@ xfs_report_verify_errors(
>  	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
>  }
>  
> -/* Verify disk blocks with GETFSMAP */
> -
> -struct xfs_verify_extent {
> -	struct read_verify_pool	*readverify;
> -	struct bitmap		*d_bad;		/* bytes */
> -	struct bitmap		*r_bad;		/* bytes */
> -};
> -
>  /* Report an IO error resulting from read-verify based off getfsmap. */
>  static bool
>  xfs_check_rmap_error_report(
> @@ -393,7 +396,9 @@ xfs_check_rmap(
>  	void				*arg)
>  {
>  	struct xfs_verify_extent	*ve = arg;
> -	struct disk			*disk;
> +	struct read_verify_pool		*rvp;
> +
> +	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
>  
>  	dbg_printf("rmap dev %d:%d phys %"PRIu64" owner %"PRId64
>  			" offset %"PRIu64" len %"PRIu64" flags 0x%x\n",
> @@ -420,19 +425,32 @@ xfs_check_rmap(
>  	/* XXX: Filter out directory data blocks. */
>  
>  	/* Schedule the read verify command for (eventual) running. */
> -	disk = xfs_dev_to_disk(ctx, map->fmr_device);
> -
> -	read_verify_schedule_io(ve->readverify, disk, map->fmr_physical,
> -			map->fmr_length, ve);
> +	read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, ve);
>  
>  out:
>  	/* Is this the last extent?  Fire off the read. */
>  	if (map->fmr_flags & FMR_OF_LAST)
> -		read_verify_force_io(ve->readverify);
> +		read_verify_force_io(rvp);
>  
>  	return true;
>  }
>  
> +/* Wait for read/verify actions to finish, then return # bytes checked. */
> +static uint64_t
> +clean_pool(
> +	struct read_verify_pool	*rvp)
> +{
> +	uint64_t		ret;
> +
> +	if (!rvp)
> +		return 0;
> +
> +	read_verify_pool_flush(rvp);
> +	ret = read_verify_bytes(rvp);
> +	read_verify_pool_destroy(rvp);
> +	return ret;
> +}
> +
>  /*
>   * Read verify all the file data blocks in a filesystem.  Since XFS doesn't
>   * do data checksums, we trust that the underlying storage will pass back
> @@ -445,7 +463,7 @@ bool
>  xfs_scan_blocks(
>  	struct scrub_ctx		*ctx)
>  {
> -	struct xfs_verify_extent	ve;
> +	struct xfs_verify_extent	ve = { NULL };
>  	bool				moveon;
>  
>  	moveon = bitmap_init(&ve.d_bad);
> @@ -460,21 +478,43 @@ xfs_scan_blocks(
>  		goto out_dbad;
>  	}
>  
> -	ve.readverify = read_verify_pool_init(ctx, ctx->geo.blocksize,
> -			xfs_check_rmap_ioerr, disk_heads(ctx->datadev),
> +	ve.rvp_data = read_verify_pool_init(ctx, ctx->datadev,
> +			ctx->geo.blocksize, xfs_check_rmap_ioerr,
>  			scrub_nproc(ctx));
> -	if (!ve.readverify) {
> +	if (!ve.rvp_data) {
>  		moveon = false;
>  		str_info(ctx, ctx->mntpoint,
> -_("Could not create media verifier."));
> +_("Could not create data device media verifier."));
>  		goto out_rbad;
>  	}
> +	if (ctx->logdev) {
> +		ve.rvp_log = read_verify_pool_init(ctx, ctx->logdev,
> +				ctx->geo.blocksize, xfs_check_rmap_ioerr,
> +				scrub_nproc(ctx));
> +		if (!ve.rvp_log) {
> +			moveon = false;
> +			str_info(ctx, ctx->mntpoint,
> +	_("Could not create log device media verifier."));
> +			goto out_datapool;
> +		}
> +	}
> +	if (ctx->rtdev) {
> +		ve.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev,
> +				ctx->geo.blocksize, xfs_check_rmap_ioerr,
> +				scrub_nproc(ctx));
> +		if (!ve.rvp_realtime) {
> +			moveon = false;
> +			str_info(ctx, ctx->mntpoint,
> +	_("Could not create realtime device media verifier."));
> +			goto out_logpool;
> +		}
> +	}
>  	moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &ve);
>  	if (!moveon)
> -		goto out_pool;
> -	read_verify_pool_flush(ve.readverify);
> -	ctx->bytes_checked += read_verify_bytes(ve.readverify);
> -	read_verify_pool_destroy(ve.readverify);
> +		goto out_rtpool;
> +	ctx->bytes_checked += clean_pool(ve.rvp_data);
> +	ctx->bytes_checked += clean_pool(ve.rvp_log);
> +	ctx->bytes_checked += clean_pool(ve.rvp_realtime);
>  
>  	/* Scan the whole dir tree to see what matches the bad extents. */
>  	if (!bitmap_empty(ve.d_bad) || !bitmap_empty(ve.r_bad))
> @@ -484,8 +524,14 @@ _("Could not create media verifier."));
>  	bitmap_free(&ve.d_bad);
>  	return moveon;
>  
> -out_pool:
> -	read_verify_pool_destroy(ve.readverify);
> +out_rtpool:
> +	if (ve.rvp_realtime)
> +		read_verify_pool_destroy(ve.rvp_realtime);
> +out_logpool:
> +	if (ve.rvp_log)
> +		read_verify_pool_destroy(ve.rvp_log);
> +out_datapool:
> +	read_verify_pool_destroy(ve.rvp_data);
>  out_rbad:
>  	bitmap_free(&ve.r_bad);
>  out_dbad:
> diff --git a/scrub/read_verify.c b/scrub/read_verify.c
> index b5774736..4a9b91f2 100644
> --- a/scrub/read_verify.c
> +++ b/scrub/read_verify.c
> @@ -50,6 +50,7 @@ struct read_verify_pool {
>  	void			*readbuf;	/* read buffer */
>  	struct ptcounter	*verified_bytes;
>  	struct ptvar		*rvstate;	/* combines read requests */
> +	struct disk		*disk;		/* which disk? */
>  	read_verify_ioerr_fn_t	ioerr_fn;	/* io error callback */
>  	size_t			miniosz;	/* minimum io size, bytes */
>  };
> @@ -57,19 +58,18 @@ struct read_verify_pool {
>  /*
>   * Create a thread pool to run read verifiers.
>   *
> + * @disk is the disk we want to verify.
>   * @miniosz is the minimum size of an IO to expect (in bytes).
>   * @ioerr_fn will be called when IO errors occur.
> - * @nproc is the maximum number of verify requests that may be sent to a disk
> - * at any given time.
>   * @submitter_threads is the number of threads that may be sending verify
>   * requests at any given time.
>   */
>  struct read_verify_pool *
>  read_verify_pool_init(
>  	struct scrub_ctx		*ctx,
> +	struct disk			*disk,
>  	size_t				miniosz,
>  	read_verify_ioerr_fn_t		ioerr_fn,
> -	unsigned int			nproc,
>  	unsigned int			submitter_threads)
>  {
>  	struct read_verify_pool		*rvp;
> @@ -89,6 +89,7 @@ read_verify_pool_init(
>  		goto out_buf;
>  	rvp->miniosz = miniosz;
>  	rvp->ctx = ctx;
> +	rvp->disk = disk;
>  	rvp->ioerr_fn = ioerr_fn;
>  	rvp->rvstate = ptvar_init(submitter_threads,
>  			sizeof(struct read_verify));
> @@ -97,7 +98,8 @@ read_verify_pool_init(
>  	/* Run in the main thread if we only want one thread. */
>  	if (nproc == 1)
>  		nproc = 0;
> -	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp, nproc);
> +	ret = workqueue_create(&rvp->wq, (struct xfs_mount *)rvp,
> +			disk_heads(disk));
>  	if (ret)
>  		goto out_rvstate;
>  	return rvp;
> @@ -150,17 +152,16 @@ read_verify(
>  	rvp = (struct read_verify_pool *)wq->wq_ctx;
>  	while (rv->io_length > 0) {
>  		len = min(rv->io_length, RVP_IO_MAX_SIZE);
> -		dbg_printf("diskverify %d %"PRIu64" %zu\n", rv->io_disk->d_fd,
> -				rv->io_start, len);
> -		sz = disk_read_verify(rv->io_disk, rvp->readbuf,
> +		dbg_printf("diskverify %d %"PRIu64" %zu\n", rvp->disk->d_fd,
>  				rv->io_start, len);
> +		sz = disk_read_verify(rvp->disk, rvp->readbuf, rv->io_start,
> +				len);
>  		if (sz < 0) {
>  			dbg_printf("IOERR %d %"PRIu64" %zu\n",
> -					rv->io_disk->d_fd,
> -					rv->io_start, len);
> +					rvp->disk->d_fd, rv->io_start, len);
>  			/* IO error, so try the next logical block. */
>  			len = rvp->miniosz;
> -			rvp->ioerr_fn(rvp->ctx, rv->io_disk, rv->io_start, len,
> +			rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start, len,
>  					errno, rv->io_end_arg);
>  		}
>  
> @@ -184,11 +185,11 @@ read_verify_queue(
>  	bool				ret;
>  
>  	dbg_printf("verify fd %d start %"PRIu64" len %"PRIu64"\n",
> -			rv->io_disk->d_fd, rv->io_start, rv->io_length);
> +			rvp->disk->d_fd, rv->io_start, rv->io_length);
>  
>  	tmp = malloc(sizeof(struct read_verify));
>  	if (!tmp) {
> -		rvp->ioerr_fn(rvp->ctx, rv->io_disk, rv->io_start,
> +		rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start,
>  				rv->io_length, errno, rv->io_end_arg);
>  		return true;
>  	}
> @@ -212,7 +213,6 @@ _("Could not queue read-verify work."));
>  bool
>  read_verify_schedule_io(
>  	struct read_verify_pool		*rvp,
> -	struct disk			*disk,
>  	uint64_t			start,
>  	uint64_t			length,
>  	void				*end_arg)
> @@ -231,7 +231,7 @@ read_verify_schedule_io(
>  	 * reporting is the same, and the two extents are close,
>  	 * we can combine them.
>  	 */
> -	if (rv->io_length > 0 && disk == rv->io_disk &&
> +	if (rv->io_length > 0 &&
>  	    end_arg == rv->io_end_arg &&
>  	    ((start >= rv->io_start && start <= rv_end + RVP_IO_BATCH_LOCALITY) ||
>  	     (rv->io_start >= start &&
> @@ -244,7 +244,6 @@ read_verify_schedule_io(
>  			return read_verify_queue(rvp, rv);
>  
>  		/* Stash the new IO. */
> -		rv->io_disk = disk;
>  		rv->io_start = start;
>  		rv->io_length = length;
>  		rv->io_end_arg = end_arg;
> diff --git a/scrub/read_verify.h b/scrub/read_verify.h
> index 1e7fd83f..5fabe5e0 100644
> --- a/scrub/read_verify.h
> +++ b/scrub/read_verify.h
> @@ -8,6 +8,7 @@
>  
>  struct scrub_ctx;
>  struct read_verify_pool;
> +struct disk;
>  
>  /* Function called when an IO error happens. */
>  typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
> @@ -15,13 +16,14 @@ typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx,
>  		int error, void *arg);
>  
>  struct read_verify_pool *read_verify_pool_init(struct scrub_ctx *ctx,
> -		size_t miniosz, read_verify_ioerr_fn_t ioerr_fn,
> -		unsigned int nproc, unsigned int submitter_threads);
> +		struct disk *disk, size_t miniosz,
> +		read_verify_ioerr_fn_t ioerr_fn,
> +		unsigned int submitter_threads);
>  void read_verify_pool_flush(struct read_verify_pool *rvp);
>  void read_verify_pool_destroy(struct read_verify_pool *rvp);
>  
> -bool read_verify_schedule_io(struct read_verify_pool *rvp, struct disk *disk,
> -		uint64_t start, uint64_t length, void *end_arg);
> +bool read_verify_schedule_io(struct read_verify_pool *rvp, uint64_t start,
> +		uint64_t length, void *end_arg);
>  bool read_verify_force_io(struct read_verify_pool *rvp);
>  uint64_t read_verify_bytes(struct read_verify_pool *rvp);
>  
> 

[PATCH 16/36] xfs_scrub: don't close mnt_fd when mnt_fd open fails

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

If we fail to open the mountpoint during phase 1 of scrub, don't bother
trying to close the file descriptor since it's silly to spray error
messages about that.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase1.c |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)


diff --git a/scrub/phase1.c b/scrub/phase1.c
index 6b472147..04a5f4a9 100644
--- a/scrub/phase1.c
+++ b/scrub/phase1.c
@@ -59,9 +59,11 @@ xfs_cleanup_fs(
 	if (ctx->datadev)
 		disk_close(ctx->datadev);
 	fshandle_destroy();
-	error = close(ctx->mnt_fd);
-	if (error)
-		str_errno(ctx, _("closing mountpoint fd"));
+	if (ctx->mnt_fd >= 0) {
+		error = close(ctx->mnt_fd);
+		if (error)
+			str_errno(ctx, _("closing mountpoint fd"));
+	}
 	fs_table_destroy();
 
 	return true;

Re: [PATCH 16/36] xfs_scrub: don't close mnt_fd when mnt_fd open fails

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> If we fail to open the mountpoint during phase 1 of scrub, don't bother
> trying to close the file descriptor since it's silly to spray error
> messages about that.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/phase1.c |    8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> 
> diff --git a/scrub/phase1.c b/scrub/phase1.c
> index 6b472147..04a5f4a9 100644
> --- a/scrub/phase1.c
> +++ b/scrub/phase1.c
> @@ -59,9 +59,11 @@ xfs_cleanup_fs(
>  	if (ctx->datadev)
>  		disk_close(ctx->datadev);
>  	fshandle_destroy();
> -	error = close(ctx->mnt_fd);
> -	if (error)
> -		str_errno(ctx, _("closing mountpoint fd"));
> +	if (ctx->mnt_fd >= 0) {
> +		error = close(ctx->mnt_fd);
> +		if (error)
> +			str_errno(ctx, _("closing mountpoint fd"));
> +	}
>  	fs_table_destroy();
>  
>  	return true;
> 

[PATCH 17/36] xfs_scrub: check label for misleading characters

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Make sure that we can retrieve the label and that it doesn't contain
anything potentially misleading.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase5.c   |   54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 scrub/unicrash.c |   24 ++++++++++++++++++++++++
 scrub/unicrash.h |    5 +++++
 3 files changed, 83 insertions(+)


diff --git a/scrub/phase5.c b/scrub/phase5.c
index 6ffcec2d..49886e6f 100644
--- a/scrub/phase5.c
+++ b/scrub/phase5.c
@@ -11,6 +11,7 @@
 #ifdef HAVE_LIBATTR
 # include <attr/attributes.h>
 #endif
+#include <linux/fs.h>
 #include "handle.h"
 #include "list.h"
 #include "path.h"
@@ -282,6 +283,55 @@ xfs_scrub_connections(
 	return *pmoveon ? 0 : XFS_ITERATE_INODES_ABORT;
 }
 
+#ifndef FS_IOC_GETFSLABEL
+# define FSLABEL_MAX		256
+# define FS_IOC_GETFSLABEL	_IOR(0x94, 49, char[FSLABEL_MAX])
+#endif /* FS_IOC_GETFSLABEL */
+
+/*
+ * Check the filesystem label for Unicode normalization problems or misleading
+ * sequences.
+ */
+static bool
+xfs_scrub_fs_label(
+	struct scrub_ctx		*ctx)
+{
+	char				label[FSLABEL_MAX];
+	struct unicrash			*uc = NULL;
+	bool				moveon = true;
+	int				error;
+
+	moveon = unicrash_fs_label_init(&uc, ctx);
+	if (!moveon)
+		return false;
+
+	/* Retrieve label; quietly bail if we don't support that. */
+	error = ioctl(ctx->mnt_fd, FS_IOC_GETFSLABEL, &label);
+	if (error) {
+		if (errno != EOPNOTSUPP && errno != ENOTTY) {
+			moveon = false;
+			perror(ctx->mntpoint);
+		}
+		goto out;
+	}
+
+	/* Ignore empty labels. */
+	if (label[0] == 0)
+		goto out;
+
+	/* Otherwise check for weirdness. */
+	if (uc)
+		moveon = unicrash_check_fs_label(uc, ctx->mntpoint, label);
+	else
+		moveon = xfs_scrub_check_name(ctx, ctx->mntpoint,
+				_("filesystem label"), label);
+	if (!moveon)
+		goto out;
+out:
+	unicrash_free(uc);
+	return moveon;
+}
+
 /* Check directory connectivity. */
 bool
 xfs_scan_connections(
@@ -296,6 +346,10 @@ _("Filesystem has errors, skipping connectivity checks."));
 		return true;
 	}
 
+	moveon = xfs_scrub_fs_label(ctx);
+	if (!moveon)
+		return false;
+
 	ret = xfs_scan_all_inodes(ctx, xfs_scrub_connections, &moveon);
 	if (!ret)
 		moveon = false;
diff --git a/scrub/unicrash.c b/scrub/unicrash.c
index a95fc305..121eedbc 100644
--- a/scrub/unicrash.c
+++ b/scrub/unicrash.c
@@ -465,6 +465,15 @@ unicrash_xattr_init(
 			is_only_root_writable(bstat));
 }
 
+/* Initialize the collision detector for a filesystem label. */
+bool
+unicrash_fs_label_init(
+	struct unicrash		**ucp,
+	struct scrub_ctx	*ctx)
+{
+	return unicrash_init(ucp, ctx, false, 16, true);
+}
+
 /* Free the crash detector. */
 void
 unicrash_free(
@@ -698,3 +707,18 @@ unicrash_check_xattr_name(
 	return __unicrash_check_name(uc, descr, _("extended attribute"),
 			attrname, 0);
 }
+
+/*
+ * Check the fs label for unicode normalization problems or misleading bits.
+ */
+bool
+unicrash_check_fs_label(
+	struct unicrash		*uc,
+	const char		*descr,
+	const char		*label)
+{
+	if (!uc)
+		return true;
+	return __unicrash_check_name(uc, descr, _("filesystem label"),
+			label, 0);
+}
diff --git a/scrub/unicrash.h b/scrub/unicrash.h
index 7d7276a8..85fcabc6 100644
--- a/scrub/unicrash.h
+++ b/scrub/unicrash.h
@@ -17,17 +17,22 @@ bool unicrash_dir_init(struct unicrash **ucp, struct scrub_ctx *ctx,
 		struct xfs_bstat *bstat);
 bool unicrash_xattr_init(struct unicrash **ucp, struct scrub_ctx *ctx,
 		struct xfs_bstat *bstat);
+bool unicrash_fs_label_init(struct unicrash **ucp, struct scrub_ctx *ctx);
 void unicrash_free(struct unicrash *uc);
 bool unicrash_check_dir_name(struct unicrash *uc, const char *descr,
 		struct dirent *dirent);
 bool unicrash_check_xattr_name(struct unicrash *uc, const char *descr,
 		const char *attrname);
+bool unicrash_check_fs_label(struct unicrash *uc, const char *descr,
+		const char *label);
 #else
 # define unicrash_dir_init(u, c, b)		(true)
 # define unicrash_xattr_init(u, c, b)		(true)
+# define unicrash_label_init(u, c)		(true)
 # define unicrash_free(u)			do {(u) = (u);} while (0)
 # define unicrash_check_dir_name(u, d, n)	(true)
 # define unicrash_check_xattr_name(u, d, n)	(true)
+# define unicrash_check_fs_label(u, d, n)	(true)
 #endif /* HAVE_LIBICU */
 
 #endif /* XFS_SCRUB_UNICRASH_H_ */

Re: [PATCH 17/36] xfs_scrub: check label for misleading characters

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Make sure that we can retrieve the label and that it doesn't contain
> anything potentially misleading.

I still don't know for sure what the risk is here of having weird
chars in a label, but sure?  :)  Anyway, nitpick below.

> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  scrub/phase5.c   |   54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  scrub/unicrash.c |   24 ++++++++++++++++++++++++
>  scrub/unicrash.h |    5 +++++
>  3 files changed, 83 insertions(+)
> 
> 
> diff --git a/scrub/phase5.c b/scrub/phase5.c
> index 6ffcec2d..49886e6f 100644
> --- a/scrub/phase5.c
> +++ b/scrub/phase5.c
> @@ -11,6 +11,7 @@
>  #ifdef HAVE_LIBATTR
>  # include <attr/attributes.h>
>  #endif
> +#include <linux/fs.h>
>  #include "handle.h"
>  #include "list.h"
>  #include "path.h"
> @@ -282,6 +283,55 @@ xfs_scrub_connections(
>  	return *pmoveon ? 0 : XFS_ITERATE_INODES_ABORT;
>  }
>  
> +#ifndef FS_IOC_GETFSLABEL
> +# define FSLABEL_MAX		256
> +# define FS_IOC_GETFSLABEL	_IOR(0x94, 49, char[FSLABEL_MAX])
> +#endif /* FS_IOC_GETFSLABEL */
> +
> +/*
> + * Check the filesystem label for Unicode normalization problems or misleading
> + * sequences.
> + */
> +static bool
> +xfs_scrub_fs_label(
> +	struct scrub_ctx		*ctx)
> +{
> +	char				label[FSLABEL_MAX];
> +	struct unicrash			*uc = NULL;
> +	bool				moveon = true;
> +	int				error;
> +
> +	moveon = unicrash_fs_label_init(&uc, ctx);
> +	if (!moveon)
> +		return false;
> +
> +	/* Retrieve label; quietly bail if we don't support that. */
> +	error = ioctl(ctx->mnt_fd, FS_IOC_GETFSLABEL, &label);
> +	if (error) {
> +		if (errno != EOPNOTSUPP && errno != ENOTTY) {
> +			moveon = false;
> +			perror(ctx->mntpoint);
> +		}
> +		goto out;
> +	}
> +
> +	/* Ignore empty labels. */
> +	if (label[0] == 0)
> +		goto out;
> +
> +	/* Otherwise check for weirdness. */
> +	if (uc)
> +		moveon = unicrash_check_fs_label(uc, ctx->mntpoint, label);
> +	else
> +		moveon = xfs_scrub_check_name(ctx, ctx->mntpoint,
> +				_("filesystem label"), label);
> +	if (!moveon)
> +		goto out;

This test & goto ^^^ seems rather pointless...

> +out:
> +	unicrash_free(uc);
> +	return moveon;
> +}
> +
>  /* Check directory connectivity. */
>  bool
>  xfs_scan_connections(
> @@ -296,6 +346,10 @@ _("Filesystem has errors, skipping connectivity checks."));
>  		return true;
>  	}
>  
> +	moveon = xfs_scrub_fs_label(ctx);
> +	if (!moveon)
> +		return false;
> +
>  	ret = xfs_scan_all_inodes(ctx, xfs_scrub_connections, &moveon);
>  	if (!ret)
>  		moveon = false;
> diff --git a/scrub/unicrash.c b/scrub/unicrash.c
> index a95fc305..121eedbc 100644
> --- a/scrub/unicrash.c
> +++ b/scrub/unicrash.c
> @@ -465,6 +465,15 @@ unicrash_xattr_init(
>  			is_only_root_writable(bstat));
>  }
>  
> +/* Initialize the collision detector for a filesystem label. */
> +bool
> +unicrash_fs_label_init(
> +	struct unicrash		**ucp,
> +	struct scrub_ctx	*ctx)
> +{
> +	return unicrash_init(ucp, ctx, false, 16, true);
> +}
> +
>  /* Free the crash detector. */
>  void
>  unicrash_free(
> @@ -698,3 +707,18 @@ unicrash_check_xattr_name(
>  	return __unicrash_check_name(uc, descr, _("extended attribute"),
>  			attrname, 0);
>  }
> +
> +/*
> + * Check the fs label for unicode normalization problems or misleading bits.
> + */
> +bool
> +unicrash_check_fs_label(
> +	struct unicrash		*uc,
> +	const char		*descr,
> +	const char		*label)
> +{
> +	if (!uc)
> +		return true;
> +	return __unicrash_check_name(uc, descr, _("filesystem label"),
> +			label, 0);
> +}
> diff --git a/scrub/unicrash.h b/scrub/unicrash.h
> index 7d7276a8..85fcabc6 100644
> --- a/scrub/unicrash.h
> +++ b/scrub/unicrash.h
> @@ -17,17 +17,22 @@ bool unicrash_dir_init(struct unicrash **ucp, struct scrub_ctx *ctx,
>  		struct xfs_bstat *bstat);
>  bool unicrash_xattr_init(struct unicrash **ucp, struct scrub_ctx *ctx,
>  		struct xfs_bstat *bstat);
> +bool unicrash_fs_label_init(struct unicrash **ucp, struct scrub_ctx *ctx);
>  void unicrash_free(struct unicrash *uc);
>  bool unicrash_check_dir_name(struct unicrash *uc, const char *descr,
>  		struct dirent *dirent);
>  bool unicrash_check_xattr_name(struct unicrash *uc, const char *descr,
>  		const char *attrname);
> +bool unicrash_check_fs_label(struct unicrash *uc, const char *descr,
> +		const char *label);
>  #else
>  # define unicrash_dir_init(u, c, b)		(true)
>  # define unicrash_xattr_init(u, c, b)		(true)
> +# define unicrash_label_init(u, c)		(true)
>  # define unicrash_free(u)			do {(u) = (u);} while (0)
>  # define unicrash_check_dir_name(u, d, n)	(true)
>  # define unicrash_check_xattr_name(u, d, n)	(true)
> +# define unicrash_check_fs_label(u, d, n)	(true)
>  #endif /* HAVE_LIBICU */
>  
>  #endif /* XFS_SCRUB_UNICRASH_H_ */
> 

Re: [PATCH 17/36] xfs_scrub: check label for misleading characters

[Darrick J. Wong]

On Wed, Mar 20, 2019 at 03:09:56PM -0500, Eric Sandeen wrote:
> On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Make sure that we can retrieve the label and that it doesn't contain
> > anything potentially misleading.
> 
> I still don't know for sure what the risk is here of having weird
> chars in a label, but sure?  :)  Anyway, nitpick below.
> 
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  scrub/phase5.c   |   54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  scrub/unicrash.c |   24 ++++++++++++++++++++++++
> >  scrub/unicrash.h |    5 +++++
> >  3 files changed, 83 insertions(+)
> > 
> > 
> > diff --git a/scrub/phase5.c b/scrub/phase5.c
> > index 6ffcec2d..49886e6f 100644
> > --- a/scrub/phase5.c
> > +++ b/scrub/phase5.c
> > @@ -11,6 +11,7 @@
> >  #ifdef HAVE_LIBATTR
> >  # include <attr/attributes.h>
> >  #endif
> > +#include <linux/fs.h>
> >  #include "handle.h"
> >  #include "list.h"
> >  #include "path.h"
> > @@ -282,6 +283,55 @@ xfs_scrub_connections(
> >  	return *pmoveon ? 0 : XFS_ITERATE_INODES_ABORT;
> >  }
> >  
> > +#ifndef FS_IOC_GETFSLABEL
> > +# define FSLABEL_MAX		256
> > +# define FS_IOC_GETFSLABEL	_IOR(0x94, 49, char[FSLABEL_MAX])
> > +#endif /* FS_IOC_GETFSLABEL */
> > +
> > +/*
> > + * Check the filesystem label for Unicode normalization problems or misleading
> > + * sequences.
> > + */
> > +static bool
> > +xfs_scrub_fs_label(
> > +	struct scrub_ctx		*ctx)
> > +{
> > +	char				label[FSLABEL_MAX];
> > +	struct unicrash			*uc = NULL;
> > +	bool				moveon = true;
> > +	int				error;
> > +
> > +	moveon = unicrash_fs_label_init(&uc, ctx);
> > +	if (!moveon)
> > +		return false;
> > +
> > +	/* Retrieve label; quietly bail if we don't support that. */
> > +	error = ioctl(ctx->mnt_fd, FS_IOC_GETFSLABEL, &label);
> > +	if (error) {
> > +		if (errno != EOPNOTSUPP && errno != ENOTTY) {
> > +			moveon = false;
> > +			perror(ctx->mntpoint);
> > +		}
> > +		goto out;
> > +	}
> > +
> > +	/* Ignore empty labels. */
> > +	if (label[0] == 0)
> > +		goto out;
> > +
> > +	/* Otherwise check for weirdness. */
> > +	if (uc)
> > +		moveon = unicrash_check_fs_label(uc, ctx->mntpoint, label);
> > +	else
> > +		moveon = xfs_scrub_check_name(ctx, ctx->mntpoint,
> > +				_("filesystem label"), label);
> > +	if (!moveon)
> > +		goto out;
> 
> This test & goto ^^^ seems rather pointless...

Yes, it's currently pointless seeing as there's nothing else that goes
on before the label.  I didn't want to leave the logic bomb that if
anyone ever /does/ add something here they'll have to add back the
if-goto-out part.

/me shrugs, will take it out and resubmit if you like. :)

--D

> > +out:
> > +	unicrash_free(uc);
> > +	return moveon;
> > +}
> > +
> >  /* Check directory connectivity. */
> >  bool
> >  xfs_scan_connections(
> > @@ -296,6 +346,10 @@ _("Filesystem has errors, skipping connectivity checks."));
> >  		return true;
> >  	}
> >  
> > +	moveon = xfs_scrub_fs_label(ctx);
> > +	if (!moveon)
> > +		return false;
> > +
> >  	ret = xfs_scan_all_inodes(ctx, xfs_scrub_connections, &moveon);
> >  	if (!ret)
> >  		moveon = false;
> > diff --git a/scrub/unicrash.c b/scrub/unicrash.c
> > index a95fc305..121eedbc 100644
> > --- a/scrub/unicrash.c
> > +++ b/scrub/unicrash.c
> > @@ -465,6 +465,15 @@ unicrash_xattr_init(
> >  			is_only_root_writable(bstat));
> >  }
> >  
> > +/* Initialize the collision detector for a filesystem label. */
> > +bool
> > +unicrash_fs_label_init(
> > +	struct unicrash		**ucp,
> > +	struct scrub_ctx	*ctx)
> > +{
> > +	return unicrash_init(ucp, ctx, false, 16, true);
> > +}
> > +
> >  /* Free the crash detector. */
> >  void
> >  unicrash_free(
> > @@ -698,3 +707,18 @@ unicrash_check_xattr_name(
> >  	return __unicrash_check_name(uc, descr, _("extended attribute"),
> >  			attrname, 0);
> >  }
> > +
> > +/*
> > + * Check the fs label for unicode normalization problems or misleading bits.
> > + */
> > +bool
> > +unicrash_check_fs_label(
> > +	struct unicrash		*uc,
> > +	const char		*descr,
> > +	const char		*label)
> > +{
> > +	if (!uc)
> > +		return true;
> > +	return __unicrash_check_name(uc, descr, _("filesystem label"),
> > +			label, 0);
> > +}
> > diff --git a/scrub/unicrash.h b/scrub/unicrash.h
> > index 7d7276a8..85fcabc6 100644
> > --- a/scrub/unicrash.h
> > +++ b/scrub/unicrash.h
> > @@ -17,17 +17,22 @@ bool unicrash_dir_init(struct unicrash **ucp, struct scrub_ctx *ctx,
> >  		struct xfs_bstat *bstat);
> >  bool unicrash_xattr_init(struct unicrash **ucp, struct scrub_ctx *ctx,
> >  		struct xfs_bstat *bstat);
> > +bool unicrash_fs_label_init(struct unicrash **ucp, struct scrub_ctx *ctx);
> >  void unicrash_free(struct unicrash *uc);
> >  bool unicrash_check_dir_name(struct unicrash *uc, const char *descr,
> >  		struct dirent *dirent);
> >  bool unicrash_check_xattr_name(struct unicrash *uc, const char *descr,
> >  		const char *attrname);
> > +bool unicrash_check_fs_label(struct unicrash *uc, const char *descr,
> > +		const char *label);
> >  #else
> >  # define unicrash_dir_init(u, c, b)		(true)
> >  # define unicrash_xattr_init(u, c, b)		(true)
> > +# define unicrash_label_init(u, c)		(true)
> >  # define unicrash_free(u)			do {(u) = (u);} while (0)
> >  # define unicrash_check_dir_name(u, d, n)	(true)
> >  # define unicrash_check_xattr_name(u, d, n)	(true)
> > +# define unicrash_check_fs_label(u, d, n)	(true)
> >  #endif /* HAVE_LIBICU */
> >  
> >  #endif /* XFS_SCRUB_UNICRASH_H_ */
> > 

Re: [PATCH 17/36] xfs_scrub: check label for misleading characters

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Make sure that we can retrieve the label and that it doesn't contain
> anything potentially misleading.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Ok nitpick withdrawn

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/phase5.c   |   54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  scrub/unicrash.c |   24 ++++++++++++++++++++++++
>  scrub/unicrash.h |    5 +++++
>  3 files changed, 83 insertions(+)
> 
> 
> diff --git a/scrub/phase5.c b/scrub/phase5.c
> index 6ffcec2d..49886e6f 100644
> --- a/scrub/phase5.c
> +++ b/scrub/phase5.c
> @@ -11,6 +11,7 @@
>  #ifdef HAVE_LIBATTR
>  # include <attr/attributes.h>
>  #endif
> +#include <linux/fs.h>
>  #include "handle.h"
>  #include "list.h"
>  #include "path.h"
> @@ -282,6 +283,55 @@ xfs_scrub_connections(
>  	return *pmoveon ? 0 : XFS_ITERATE_INODES_ABORT;
>  }
>  
> +#ifndef FS_IOC_GETFSLABEL
> +# define FSLABEL_MAX		256
> +# define FS_IOC_GETFSLABEL	_IOR(0x94, 49, char[FSLABEL_MAX])
> +#endif /* FS_IOC_GETFSLABEL */
> +
> +/*
> + * Check the filesystem label for Unicode normalization problems or misleading
> + * sequences.
> + */
> +static bool
> +xfs_scrub_fs_label(
> +	struct scrub_ctx		*ctx)
> +{
> +	char				label[FSLABEL_MAX];
> +	struct unicrash			*uc = NULL;
> +	bool				moveon = true;
> +	int				error;
> +
> +	moveon = unicrash_fs_label_init(&uc, ctx);
> +	if (!moveon)
> +		return false;
> +
> +	/* Retrieve label; quietly bail if we don't support that. */
> +	error = ioctl(ctx->mnt_fd, FS_IOC_GETFSLABEL, &label);
> +	if (error) {
> +		if (errno != EOPNOTSUPP && errno != ENOTTY) {
> +			moveon = false;
> +			perror(ctx->mntpoint);
> +		}
> +		goto out;
> +	}
> +
> +	/* Ignore empty labels. */
> +	if (label[0] == 0)
> +		goto out;
> +
> +	/* Otherwise check for weirdness. */
> +	if (uc)
> +		moveon = unicrash_check_fs_label(uc, ctx->mntpoint, label);
> +	else
> +		moveon = xfs_scrub_check_name(ctx, ctx->mntpoint,
> +				_("filesystem label"), label);
> +	if (!moveon)
> +		goto out;
> +out:
> +	unicrash_free(uc);
> +	return moveon;
> +}
> +
>  /* Check directory connectivity. */
>  bool
>  xfs_scan_connections(
> @@ -296,6 +346,10 @@ _("Filesystem has errors, skipping connectivity checks."));
>  		return true;
>  	}
>  
> +	moveon = xfs_scrub_fs_label(ctx);
> +	if (!moveon)
> +		return false;
> +
>  	ret = xfs_scan_all_inodes(ctx, xfs_scrub_connections, &moveon);
>  	if (!ret)
>  		moveon = false;
> diff --git a/scrub/unicrash.c b/scrub/unicrash.c
> index a95fc305..121eedbc 100644
> --- a/scrub/unicrash.c
> +++ b/scrub/unicrash.c
> @@ -465,6 +465,15 @@ unicrash_xattr_init(
>  			is_only_root_writable(bstat));
>  }
>  
> +/* Initialize the collision detector for a filesystem label. */
> +bool
> +unicrash_fs_label_init(
> +	struct unicrash		**ucp,
> +	struct scrub_ctx	*ctx)
> +{
> +	return unicrash_init(ucp, ctx, false, 16, true);
> +}
> +
>  /* Free the crash detector. */
>  void
>  unicrash_free(
> @@ -698,3 +707,18 @@ unicrash_check_xattr_name(
>  	return __unicrash_check_name(uc, descr, _("extended attribute"),
>  			attrname, 0);
>  }
> +
> +/*
> + * Check the fs label for unicode normalization problems or misleading bits.
> + */
> +bool
> +unicrash_check_fs_label(
> +	struct unicrash		*uc,
> +	const char		*descr,
> +	const char		*label)
> +{
> +	if (!uc)
> +		return true;
> +	return __unicrash_check_name(uc, descr, _("filesystem label"),
> +			label, 0);
> +}
> diff --git a/scrub/unicrash.h b/scrub/unicrash.h
> index 7d7276a8..85fcabc6 100644
> --- a/scrub/unicrash.h
> +++ b/scrub/unicrash.h
> @@ -17,17 +17,22 @@ bool unicrash_dir_init(struct unicrash **ucp, struct scrub_ctx *ctx,
>  		struct xfs_bstat *bstat);
>  bool unicrash_xattr_init(struct unicrash **ucp, struct scrub_ctx *ctx,
>  		struct xfs_bstat *bstat);
> +bool unicrash_fs_label_init(struct unicrash **ucp, struct scrub_ctx *ctx);
>  void unicrash_free(struct unicrash *uc);
>  bool unicrash_check_dir_name(struct unicrash *uc, const char *descr,
>  		struct dirent *dirent);
>  bool unicrash_check_xattr_name(struct unicrash *uc, const char *descr,
>  		const char *attrname);
> +bool unicrash_check_fs_label(struct unicrash *uc, const char *descr,
> +		const char *label);
>  #else
>  # define unicrash_dir_init(u, c, b)		(true)
>  # define unicrash_xattr_init(u, c, b)		(true)
> +# define unicrash_label_init(u, c)		(true)
>  # define unicrash_free(u)			do {(u) = (u);} while (0)
>  # define unicrash_check_dir_name(u, d, n)	(true)
>  # define unicrash_check_xattr_name(u, d, n)	(true)
> +# define unicrash_check_fs_label(u, d, n)	(true)
>  #endif /* HAVE_LIBICU */
>  
>  #endif /* XFS_SCRUB_UNICRASH_H_ */
> 

[PATCH 18/36] scrub: fix Makefile targets which depend on builddefs

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add Makefile dependencies for targets that require variables set in
builddefs.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/Makefile |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)


diff --git a/scrub/Makefile b/scrub/Makefile
index 6e155c2c..bbcfe338 100644
--- a/scrub/Makefile
+++ b/scrub/Makefile
@@ -3,7 +3,8 @@
 #
 
 TOPDIR = ..
-include $(TOPDIR)/include/builddefs
+builddefs=$(TOPDIR)/include/builddefs
+include $(builddefs)
 
 # On linux we get fsmap from the system or define it ourselves
 # so include this based on platform type.  If this reverts to only
@@ -103,27 +104,27 @@ LDIRT = $(XFS_SCRUB_ALL_PROG) *.service *.cron
 
 default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(OPTIONAL_TARGETS)
 
-xfs_scrub_all: xfs_scrub_all.in
+xfs_scrub_all: xfs_scrub_all.in $(builddefs)
 	@echo "    [SED]    $@"
 	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \
 		   -e "s|@pkg_version@|$(PKG_VERSION)|g" \
 		   -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@
 	$(Q)chmod a+x $@
 
-phase5.o unicrash.o xfs.o: $(TOPDIR)/include/builddefs
+phase5.o unicrash.o xfs.o: $(builddefs)
 
 include $(BUILDRULES)
 
 install: $(INSTALL_SCRUB)
 
-%.service: %.service.in
+%.service: %.service.in $(builddefs)
 	@echo "    [SED]    $@"
 	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \
 		   -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \
 		   -e "s|@pkg_lib_dir@|$(PKG_LIB_DIR)|g" \
 		   -e "s|@pkg_name@|$(PKG_NAME)|g" < $< > $@
 
-%.cron: %.cron.in
+%.cron: %.cron.in $(builddefs)
 	@echo "    [SED]    $@"
 	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" < $< > $@
 

Re: [PATCH 18/36] scrub: fix Makefile targets which depend on builddefs

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Add Makefile dependencies for targets that require variables set in
> builddefs.

<wonders why this is unique to scrub/ and whether we should make
scrub/ special?>

Ok, Darrick points out that scrub is the only(?) subdir that actually
substitutes builddefs variables in the /build/ process, so it is unique in
that regard.

However, builddefs also defines all the HAVE_FOO and even (as Darrick
points out) CC= so I'm still feeling like this shouldn't be a one-off
for the scrub subdir.  Thoughts?

-Eric

> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  scrub/Makefile |   11 ++++++-----
>  1 file changed, 6 insertions(+), 5 deletions(-)
> 
> 
> diff --git a/scrub/Makefile b/scrub/Makefile
> index 6e155c2c..bbcfe338 100644
> --- a/scrub/Makefile
> +++ b/scrub/Makefile
> @@ -3,7 +3,8 @@
>  #
>  
>  TOPDIR = ..
> -include $(TOPDIR)/include/builddefs
> +builddefs=$(TOPDIR)/include/builddefs
> +include $(builddefs)
>  
>  # On linux we get fsmap from the system or define it ourselves
>  # so include this based on platform type.  If this reverts to only
> @@ -103,27 +104,27 @@ LDIRT = $(XFS_SCRUB_ALL_PROG) *.service *.cron
>  
>  default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(OPTIONAL_TARGETS)
>  
> -xfs_scrub_all: xfs_scrub_all.in
> +xfs_scrub_all: xfs_scrub_all.in $(builddefs)
>  	@echo "    [SED]    $@"
>  	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \
>  		   -e "s|@pkg_version@|$(PKG_VERSION)|g" \
>  		   -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@
>  	$(Q)chmod a+x $@
>  
> -phase5.o unicrash.o xfs.o: $(TOPDIR)/include/builddefs
> +phase5.o unicrash.o xfs.o: $(builddefs)
>  
>  include $(BUILDRULES)
>  
>  install: $(INSTALL_SCRUB)
>  
> -%.service: %.service.in
> +%.service: %.service.in $(builddefs)
>  	@echo "    [SED]    $@"
>  	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \
>  		   -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \
>  		   -e "s|@pkg_lib_dir@|$(PKG_LIB_DIR)|g" \
>  		   -e "s|@pkg_name@|$(PKG_NAME)|g" < $< > $@
>  
> -%.cron: %.cron.in
> +%.cron: %.cron.in $(builddefs)
>  	@echo "    [SED]    $@"
>  	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" < $< > $@
>  
> 

Re: [PATCH 18/36] scrub: fix Makefile targets which depend on builddefs

[Darrick J. Wong]

On Wed, Mar 20, 2019 at 03:23:29PM -0500, Eric Sandeen wrote:
> On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Add Makefile dependencies for targets that require variables set in
> > builddefs.
> 
> <wonders why this is unique to scrub/ and whether we should make
> scrub/ special?>
> 
> Ok, Darrick points out that scrub is the only(?) subdir that actually
> substitutes builddefs variables in the /build/ process, so it is unique in
> that regard.
> 
> However, builddefs also defines all the HAVE_FOO and even (as Darrick
> points out) CC= so I'm still feeling like this shouldn't be a one-off
> for the scrub subdir.  Thoughts?

[Summarizing IRC chatter last night]

Dave suggested using AC_CONFIG_FILES to replace the sed-using targets in
scrub/Makefile, but it turns out that AC_CONFIG_FILES only does a single
level of substitution (because that's all that's needed for Makefiles)
so if we have the statement:

@sbindir@/xfs_scrub -foo

and variables:

exec_prefix="/usr"
sbindir="${exec_prefix}/sbin"

Then AC_CONFIG_FILES emits:

${exec_prefix}/sbin/xfs_scrub -foo

Not the fully variable-expanded absolute path we were expecting:

/usr/sbin/xfs_scrub -foo

like what systemd service files and cronjob filess are supposed to have.
That's why we need the sed commands in the Makefile and the explicit
dependency on builddefs.

FWIW the AC_CONFIG_FILES documentation implies it's only supposed to be
used for generating Makefiles, not actual build targets.

--D

> -Eric
> 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  scrub/Makefile |   11 ++++++-----
> >  1 file changed, 6 insertions(+), 5 deletions(-)
> > 
> > 
> > diff --git a/scrub/Makefile b/scrub/Makefile
> > index 6e155c2c..bbcfe338 100644
> > --- a/scrub/Makefile
> > +++ b/scrub/Makefile
> > @@ -3,7 +3,8 @@
> >  #
> >  
> >  TOPDIR = ..
> > -include $(TOPDIR)/include/builddefs
> > +builddefs=$(TOPDIR)/include/builddefs
> > +include $(builddefs)
> >  
> >  # On linux we get fsmap from the system or define it ourselves
> >  # so include this based on platform type.  If this reverts to only
> > @@ -103,27 +104,27 @@ LDIRT = $(XFS_SCRUB_ALL_PROG) *.service *.cron
> >  
> >  default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(OPTIONAL_TARGETS)
> >  
> > -xfs_scrub_all: xfs_scrub_all.in
> > +xfs_scrub_all: xfs_scrub_all.in $(builddefs)
> >  	@echo "    [SED]    $@"
> >  	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \
> >  		   -e "s|@pkg_version@|$(PKG_VERSION)|g" \
> >  		   -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@
> >  	$(Q)chmod a+x $@
> >  
> > -phase5.o unicrash.o xfs.o: $(TOPDIR)/include/builddefs
> > +phase5.o unicrash.o xfs.o: $(builddefs)
> >  
> >  include $(BUILDRULES)
> >  
> >  install: $(INSTALL_SCRUB)
> >  
> > -%.service: %.service.in
> > +%.service: %.service.in $(builddefs)
> >  	@echo "    [SED]    $@"
> >  	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \
> >  		   -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \
> >  		   -e "s|@pkg_lib_dir@|$(PKG_LIB_DIR)|g" \
> >  		   -e "s|@pkg_name@|$(PKG_NAME)|g" < $< > $@
> >  
> > -%.cron: %.cron.in
> > +%.cron: %.cron.in $(builddefs)
> >  	@echo "    [SED]    $@"
> >  	$(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" < $< > $@
> >  
> > 

[PATCH 19/36] mkfs: validate extent size hint parameters

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Validate extent and cow extent size hints that are passed to mkfs so
that we avoid formatting a filesystem that will never mount.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 mkfs/xfs_mkfs.c |   64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)


diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
index d1387ddf..9e1c6ec5 100644
--- a/mkfs/xfs_mkfs.c
+++ b/mkfs/xfs_mkfs.c
@@ -2202,6 +2202,66 @@ validate_rtextsize(
 	ASSERT(cfg->rtextblocks);
 }
 
+/* Validate the incoming extsize hint as if we were a file. */
+static void
+validate_extsize_hint(
+	struct xfs_mount	*mp,
+	struct cli_params	*cli)
+{
+	xfs_failaddr_t		fa;
+	bool			rt;
+	uint16_t		flags = 0;
+
+	rt = (cli->fsx.fsx_xflags & XFS_DIFLAG_RTINHERIT);
+
+	if (cli->fsx.fsx_xflags & XFS_DIFLAG_EXTSZINHERIT)
+		flags |= XFS_DIFLAG_EXTSIZE;
+
+	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFREG,
+			flags);
+	if (rt && fa == NULL)
+		fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize,
+				S_IFREG, flags | XFS_DIFLAG_REALTIME);
+	if (fa == NULL)
+		return;
+
+	if (rt && mp->m_sb.sb_rextsize > 1)
+		fprintf(stderr,
+	_("illegal extent size hint %lld, must be less than %u and a multiple of %u.\n"),
+				(long long)cli->fsx.fsx_extsize,
+				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2),
+				mp->m_sb.sb_rextsize);
+	else
+		fprintf(stderr,
+	_("illegal extent size hint %lld, must be less than %u.\n"),
+				(long long)cli->fsx.fsx_extsize,
+				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2));
+	usage();
+}
+
+/* Validate the incoming CoW extsize hint as if we were a file. */
+static void
+validate_cowextsize_hint(
+	struct xfs_mount	*mp,
+	struct cli_params	*cli)
+{
+	xfs_failaddr_t		fa;
+	uint64_t		flags2 = 0;
+
+	if (cli->fsx.fsx_xflags & FS_XFLAG_COWEXTSIZE)
+		flags2 |= XFS_DIFLAG2_COWEXTSIZE;
+	fa = libxfs_inode_validate_cowextsize(mp, cli->fsx.fsx_cowextsize,
+			S_IFREG, 0, flags2);
+	if (fa == NULL)
+		return;
+
+	fprintf(stderr,
+_("illegal CoW extent size hint %lld, must be less than %u.\n"),
+			(long long)cli->fsx.fsx_cowextsize,
+			min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2));
+	usage();
+}
+
 /*
  * Validate the configured stripe geometry, or is none is specified, pull
  * the configuration from the underlying device.
@@ -3945,6 +4005,10 @@ main(
 
 	finish_superblock_setup(&cfg, mp, sbp);
 
+	/* Validate the extent size hints now that @mp is fully set up. */
+	validate_extsize_hint(mp, &cli);
+	validate_cowextsize_hint(mp, &cli);
+
 	/* Print the intended geometry of the fs. */
 	if (!quiet || dry_run) {
 		struct xfs_fsop_geom	geo;

Re: [PATCH 19/36] mkfs: validate extent size hint parameters

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Validate extent and cow extent size hints that are passed to mkfs so
> that we avoid formatting a filesystem that will never mount.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  mkfs/xfs_mkfs.c |   64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 64 insertions(+)
> 
> 
> diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
> index d1387ddf..9e1c6ec5 100644
> --- a/mkfs/xfs_mkfs.c
> +++ b/mkfs/xfs_mkfs.c
> @@ -2202,6 +2202,66 @@ validate_rtextsize(
>  	ASSERT(cfg->rtextblocks);
>  }
>  
> +/* Validate the incoming extsize hint as if we were a file. */

I wonder why "as a file" when hints go on directories, right?

> +static void
> +validate_extsize_hint(
> +	struct xfs_mount	*mp,
> +	struct cli_params	*cli)
> +{
> +	xfs_failaddr_t		fa;
> +	bool			rt;
> +	uint16_t		flags = 0;
> +
> +	rt = (cli->fsx.fsx_xflags & XFS_DIFLAG_RTINHERIT);
> +
> +	if (cli->fsx.fsx_xflags & XFS_DIFLAG_EXTSZINHERIT)
> +		flags |= XFS_DIFLAG_EXTSIZE;

i.e. that feels like a weird switch from dir inherit flag to file extsize
flag, and I'm not sure why it's done here.

since the mkfs option essentially sets the inherit flag on the root inode,
(right?) I'd kind of expected this to be:

+	if (cli->fsx.fsx_xflags & XFS_DIFLAG_EXTSZINHERIT)
+		flags |= XFS_DIFLAG_EXTSZINHERIT;

(also surely fsx_xflags should contain XFLAGs? but that's another patch)

> +
> +	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFREG,
> +			flags);

and then:

+	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFDIR,
+			flags);

doesn't really matter to the validator, but conceptually this seems like a
root dir flag, not a "pretend it's a file!" flag, and seems more consistent.

> +	if (rt && fa == NULL)
> +		fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize,
> +				S_IFREG, flags | XFS_DIFLAG_REALTIME);

here too...?

We talked about this a little on irc but maybe you can remind me why you
switch to "it's a file!"

> +	if (fa == NULL)
> +		return;
> +

I wonder if there should be a comment here about keeping these calculations in
sync with xfs_inode_validate_extsize()?

> +	if (rt && mp->m_sb.sb_rextsize > 1)
> +		fprintf(stderr,
> +	_("illegal extent size hint %lld, must be less than %u and a multiple of %u.\n"),
> +				(long long)cli->fsx.fsx_extsize,
> +				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2),
> +				mp->m_sb.sb_rextsize);
> +	else
> +		fprintf(stderr,
> +	_("illegal extent size hint %lld, must be less than %u.\n"),
> +				(long long)cli->fsx.fsx_extsize,
> +				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2));
> +	usage();
> +}
> +
> +/* Validate the incoming CoW extsize hint as if we were a file. */
> +static void
> +validate_cowextsize_hint(
> +	struct xfs_mount	*mp,
> +	struct cli_params	*cli)
> +{
> +	xfs_failaddr_t		fa;
> +	uint64_t		flags2 = 0;
> +
> +	if (cli->fsx.fsx_xflags & FS_XFLAG_COWEXTSIZE)
> +		flags2 |= XFS_DIFLAG2_COWEXTSIZE;

(here the flag is the same for dir or file, but same basic question)

> +	fa = libxfs_inode_validate_cowextsize(mp, cli->fsx.fsx_cowextsize,
> +			S_IFREG, 0, flags2);
> +	if (fa == NULL)
> +		return;
> +
> +	fprintf(stderr,
> +_("illegal CoW extent size hint %lld, must be less than %u.\n"),
> +			(long long)cli->fsx.fsx_cowextsize,
> +			min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2));
> +	usage();
> +}
> +
>  /*
>   * Validate the configured stripe geometry, or is none is specified, pull
>   * the configuration from the underlying device.
> @@ -3945,6 +4005,10 @@ main(
>  
>  	finish_superblock_setup(&cfg, mp, sbp);
>  
> +	/* Validate the extent size hints now that @mp is fully set up. */
> +	validate_extsize_hint(mp, &cli);
> +	validate_cowextsize_hint(mp, &cli);
> +
>  	/* Print the intended geometry of the fs. */
>  	if (!quiet || dry_run) {
>  		struct xfs_fsop_geom	geo;
> 

Re: [PATCH 19/36] mkfs: validate extent size hint parameters

[Darrick J. Wong]

On Tue, Mar 26, 2019 at 11:59:22AM -0500, Eric Sandeen wrote:
> On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Validate extent and cow extent size hints that are passed to mkfs so
> > that we avoid formatting a filesystem that will never mount.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  mkfs/xfs_mkfs.c |   64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 64 insertions(+)
> > 
> > 
> > diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
> > index d1387ddf..9e1c6ec5 100644
> > --- a/mkfs/xfs_mkfs.c
> > +++ b/mkfs/xfs_mkfs.c
> > @@ -2202,6 +2202,66 @@ validate_rtextsize(
> >  	ASSERT(cfg->rtextblocks);
> >  }
> >  
> > +/* Validate the incoming extsize hint as if we were a file. */
> 
> I wonder why "as a file" when hints go on directories, right?

The hint is applied to the root directory with the intention of
spreading it to newly created files, so that's why I thought we should
simulate the extsize check with a file to make sure we're not formatting
a broken filesystem.

The current extsize validator code works the same between regular files
with extsize set and directories with extszinherit set, so you're right
in wondering why not simulate a directory check instead of pretending to
be a file?

My immediate answer to that is ... I had to make an arbitrary choice, so
I picked one and went with it.  A more forward-thinking answer is that
if we ever change libxfs to validate extent size hints different (or not
at all) for directories then it'd be nice to avoid leaving that logic
bomb.  Though mkfs is tightly coupled to libxfs so perhaps that's not
worth worrying about.

So, yes we could simulate the directory extsize check and then move on
to the rt file extsize if the fs is being formatted with a rt volume.
I'll change it & resubmit.

> > +static void
> > +validate_extsize_hint(
> > +	struct xfs_mount	*mp,
> > +	struct cli_params	*cli)
> > +{
> > +	xfs_failaddr_t		fa;
> > +	bool			rt;
> > +	uint16_t		flags = 0;
> > +
> > +	rt = (cli->fsx.fsx_xflags & XFS_DIFLAG_RTINHERIT);
> > +
> > +	if (cli->fsx.fsx_xflags & XFS_DIFLAG_EXTSZINHERIT)
> > +		flags |= XFS_DIFLAG_EXTSIZE;
> 
> i.e. that feels like a weird switch from dir inherit flag to file extsize
> flag, and I'm not sure why it's done here.
> 
> since the mkfs option essentially sets the inherit flag on the root inode,
> (right?) I'd kind of expected this to be:
> 
> +	if (cli->fsx.fsx_xflags & XFS_DIFLAG_EXTSZINHERIT)
> +		flags |= XFS_DIFLAG_EXTSZINHERIT;
> 
> (also surely fsx_xflags should contain XFLAGs? but that's another patch)
> 
> > +
> > +	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFREG,
> > +			flags);
> 
> and then:
> 
> +	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFDIR,
> +			flags);
> 
> doesn't really matter to the validator, but conceptually this seems like a
> root dir flag, not a "pretend it's a file!" flag, and seems more consistent.
> 
> > +	if (rt && fa == NULL)
> > +		fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize,
> > +				S_IFREG, flags | XFS_DIFLAG_REALTIME);
> 
> here too...?

This bit here checks that the extent size hint is appropriate for a
realtime file.  There's no other way to do this than to pretend to be a
file because there are no rt directories.

> We talked about this a little on irc but maybe you can remind me why you
> switch to "it's a file!"
> 
> > +	if (fa == NULL)
> > +		return;
> > +
> 
> I wonder if there should be a comment here about keeping these calculations in
> sync with xfs_inode_validate_extsize()?

Yeah.  It's too bad we can't just throw an exception containing a nice
human readable description of the exact thing we got mad about.

> > +	if (rt && mp->m_sb.sb_rextsize > 1)
> > +		fprintf(stderr,
> > +	_("illegal extent size hint %lld, must be less than %u and a multiple of %u.\n"),
> > +				(long long)cli->fsx.fsx_extsize,
> > +				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2),
> > +				mp->m_sb.sb_rextsize);
> > +	else
> > +		fprintf(stderr,
> > +	_("illegal extent size hint %lld, must be less than %u.\n"),
> > +				(long long)cli->fsx.fsx_extsize,
> > +				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2));
> > +	usage();
> > +}
> > +
> > +/* Validate the incoming CoW extsize hint as if we were a file. */
> > +static void
> > +validate_cowextsize_hint(
> > +	struct xfs_mount	*mp,
> > +	struct cli_params	*cli)
> > +{
> > +	xfs_failaddr_t		fa;
> > +	uint64_t		flags2 = 0;
> > +
> > +	if (cli->fsx.fsx_xflags & FS_XFLAG_COWEXTSIZE)
> > +		flags2 |= XFS_DIFLAG2_COWEXTSIZE;
> 
> (here the flag is the same for dir or file, but same basic question)

Same answers as above.

--D

> > +	fa = libxfs_inode_validate_cowextsize(mp, cli->fsx.fsx_cowextsize,
> > +			S_IFREG, 0, flags2);
> > +	if (fa == NULL)
> > +		return;
> > +
> > +	fprintf(stderr,
> > +_("illegal CoW extent size hint %lld, must be less than %u.\n"),
> > +			(long long)cli->fsx.fsx_cowextsize,
> > +			min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2));
> > +	usage();
> > +}
> > +
> >  /*
> >   * Validate the configured stripe geometry, or is none is specified, pull
> >   * the configuration from the underlying device.
> > @@ -3945,6 +4005,10 @@ main(
> >  
> >  	finish_superblock_setup(&cfg, mp, sbp);
> >  
> > +	/* Validate the extent size hints now that @mp is fully set up. */
> > +	validate_extsize_hint(mp, &cli);
> > +	validate_cowextsize_hint(mp, &cli);
> > +
> >  	/* Print the intended geometry of the fs. */
> >  	if (!quiet || dry_run) {
> >  		struct xfs_fsop_geom	geo;
> > 
> 
> 

[PATCH v2 19/36] mkfs: validate extent size hint parameters

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Validate extent and cow extent size hints that are passed to mkfs so
that we avoid formatting a filesystem that will never mount.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
v2: simulate extsize hint validation on a directory
---
 mkfs/xfs_mkfs.c |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)

diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
index d1387ddf..d7951faa 100644
--- a/mkfs/xfs_mkfs.c
+++ b/mkfs/xfs_mkfs.c
@@ -2202,6 +2202,85 @@ validate_rtextsize(
 	ASSERT(cfg->rtextblocks);
 }
 
+/* Validate the incoming extsize hint. */
+static void
+validate_extsize_hint(
+	struct xfs_mount	*mp,
+	struct cli_params	*cli)
+{
+	xfs_failaddr_t		fa;
+	uint16_t		flags = 0;
+
+	/*
+	 * First we validate the extent size inherit hint on a directory so
+	 * that we know that we'll be propagating a correct hint and flag to
+	 * new files on the data device.
+	 */
+	if (cli->fsx.fsx_xflags & FS_XFLAG_EXTSZINHERIT)
+		flags |= XFS_DIFLAG_EXTSZINHERIT;
+
+	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFDIR,
+			flags);
+	if (fa) {
+		fprintf(stderr,
+_("illegal extent size hint %lld, must be less than %u.\n"),
+				(long long)cli->fsx.fsx_extsize,
+				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2));
+		usage();
+	}
+
+	/*
+	 * Now we do it again with a realtime file so that we know the hint and
+	 * flag that get passed on to realtime files will be correct.
+	 */
+	if (mp->m_sb.sb_rextsize == 0)
+		return;
+
+	flags = XFS_DIFLAG_REALTIME;
+	if (cli->fsx.fsx_xflags & FS_XFLAG_EXTSZINHERIT)
+		flags |= XFS_DIFLAG_EXTSIZE;
+
+	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFREG,
+			flags);
+
+	if (fa) {
+		fprintf(stderr,
+_("illegal extent size hint %lld, must be less than %u and a multiple of %u.\n"),
+				(long long)cli->fsx.fsx_extsize,
+				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2),
+				mp->m_sb.sb_rextsize);
+		usage();
+	}
+}
+
+/* Validate the incoming CoW extsize hint. */
+static void
+validate_cowextsize_hint(
+	struct xfs_mount	*mp,
+	struct cli_params	*cli)
+{
+	xfs_failaddr_t		fa;
+	uint64_t		flags2 = 0;
+
+	/*
+	 * Validate the copy on write extent size inherit hint on a directory
+	 * so that we know that we'll be propagating a correct hint and flag to
+	 * new files on the data device.
+	 */
+	if (cli->fsx.fsx_xflags & FS_XFLAG_COWEXTSIZE)
+		flags2 |= XFS_DIFLAG2_COWEXTSIZE;
+
+	fa = libxfs_inode_validate_cowextsize(mp, cli->fsx.fsx_cowextsize,
+			S_IFDIR, 0, flags2);
+	if (fa) {
+		fprintf(stderr,
+_("illegal CoW extent size hint %lld, must be less than %u.\n"),
+				(long long)cli->fsx.fsx_cowextsize,
+				min(MAXEXTLEN, mp->m_sb.sb_agblocks / 2));
+		usage();
+	}
+}
+
 /*
  * Validate the configured stripe geometry, or is none is specified, pull
  * the configuration from the underlying device.
@@ -3945,6 +4024,10 @@ main(
 
 	finish_superblock_setup(&cfg, mp, sbp);
 
+	/* Validate the extent size hints now that @mp is fully set up. */
+	validate_extsize_hint(mp, &cli);
+	validate_cowextsize_hint(mp, &cli);
+
 	/* Print the intended geometry of the fs. */
 	if (!quiet || dry_run) {
 		struct xfs_fsop_geom	geo;

[PATCH 20/36] xfs_db: fix finobt record decoding when sparse inodes enabled

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Use the sparse inobt record field decoder (inobt_spcrc_hfld) to decode
finobt records when sparse inodes are enabled.  Otherwise, xfs_db
prints out bogus things like:

recs[1] = [startino,freecount,free]
1:[214720,16429,0xfffffffffff80000]

There can never be 16429 records in an inode btree record; instead it
should print:

recs[1] = [startino,holemask,count,freecount,free]
1:[214720,0,64,45,0xfffffffffff80000]

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 db/type.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/db/type.c b/db/type.c
index f5f65042..d42935a2 100644
--- a/db/type.c
+++ b/db/type.c
@@ -107,7 +107,7 @@ static const typ_t	__typtab_crc[] = {
 	{ TYP_SYMLINK, "symlink", handle_struct, symlink_crc_hfld,
 		&xfs_symlink_buf_ops, XFS_SYMLINK_CRC_OFF },
 	{ TYP_TEXT, "text", handle_text, NULL, NULL, TYP_F_NO_CRC_OFF },
-	{ TYP_FINOBT, "finobt", handle_struct, inobt_crc_hfld,
+	{ TYP_FINOBT, "finobt", handle_struct, inobt_spcrc_hfld,
 		&xfs_inobt_buf_ops, XFS_BTREE_SBLOCK_CRC_OFF },
 	{ TYP_NONE, NULL }
 };

Re: [PATCH 20/36] xfs_db: fix finobt record decoding when sparse inodes enabled

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Use the sparse inobt record field decoder (inobt_spcrc_hfld) to decode
> finobt records when sparse inodes are enabled.  Otherwise, xfs_db
> prints out bogus things like:
> 
> recs[1] = [startino,freecount,free]
> 1:[214720,16429,0xfffffffffff80000]
> 
> There can never be 16429 records in an inode btree record; instead it
> should print:
> 
> recs[1] = [startino,holemask,count,freecount,free]
> 1:[214720,0,64,45,0xfffffffffff80000]
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Um... :) 

commit 38c599e7782ced7d2d1b41c3182daeaddad2cee4
Author: Darrick J. Wong <darrick.wong@oracle.com>
Date:   Mon Jan 28 13:03:15 2019 -0600

    xfs_db: fix finobt record decoding when sparse inodes enabled
    
    Use the sparse inobt record field decoder (inobt_spcrc_hfld) to decode
    finobt records when sparse inodes are enabled.  Otherwise, xfs_db
    prints out bogus things like:
    
    recs[1] = [startino,freecount,free]
    1:[214720,16429,0xfffffffffff80000]
    
    There can never be 16429 records in an inode btree record; instead it
    should print:
    
    recs[1] = [startino,holemask,count,freecount,free]
    1:[214720,0,64,45,0xfffffffffff80000]
    
    Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
    Reviewed-by: Dave Chinner <dchinner@redhat.com>
    Signed-off-by: Eric Sandeen <sandeen@sandeen.net>

> ---
>  db/type.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> 
> diff --git a/db/type.c b/db/type.c
> index f5f65042..d42935a2 100644
> --- a/db/type.c
> +++ b/db/type.c
> @@ -107,7 +107,7 @@ static const typ_t	__typtab_crc[] = {
>  	{ TYP_SYMLINK, "symlink", handle_struct, symlink_crc_hfld,
>  		&xfs_symlink_buf_ops, XFS_SYMLINK_CRC_OFF },
>  	{ TYP_TEXT, "text", handle_text, NULL, NULL, TYP_F_NO_CRC_OFF },
> -	{ TYP_FINOBT, "finobt", handle_struct, inobt_crc_hfld,
> +	{ TYP_FINOBT, "finobt", handle_struct, inobt_spcrc_hfld,
>  		&xfs_inobt_buf_ops, XFS_BTREE_SBLOCK_CRC_OFF },
>  	{ TYP_NONE, NULL }
>  };
> 

Re: [PATCH 20/36] xfs_db: fix finobt record decoding when sparse inodes enabled

[Darrick J. Wong]

On Tue, Mar 26, 2019 at 12:05:04PM -0500, Eric Sandeen wrote:
> On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Use the sparse inobt record field decoder (inobt_spcrc_hfld) to decode
> > finobt records when sparse inodes are enabled.  Otherwise, xfs_db
> > prints out bogus things like:
> > 
> > recs[1] = [startino,freecount,free]
> > 1:[214720,16429,0xfffffffffff80000]
> > 
> > There can never be 16429 records in an inode btree record; instead it
> > should print:
> > 
> > recs[1] = [startino,holemask,count,freecount,free]
> > 1:[214720,0,64,45,0xfffffffffff80000]
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Um... :) 
> 
> commit 38c599e7782ced7d2d1b41c3182daeaddad2cee4
> Author: Darrick J. Wong <darrick.wong@oracle.com>
> Date:   Mon Jan 28 13:03:15 2019 -0600
> 
>     xfs_db: fix finobt record decoding when sparse inodes enabled
>     
>     Use the sparse inobt record field decoder (inobt_spcrc_hfld) to decode
>     finobt records when sparse inodes are enabled.  Otherwise, xfs_db
>     prints out bogus things like:
>     
>     recs[1] = [startino,freecount,free]
>     1:[214720,16429,0xfffffffffff80000]
>     
>     There can never be 16429 records in an inode btree record; instead it
>     should print:
>     
>     recs[1] = [startino,holemask,count,freecount,free]
>     1:[214720,0,64,45,0xfffffffffff80000]
>     
>     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
>     Reviewed-by: Dave Chinner <dchinner@redhat.com>
>     Signed-off-by: Eric Sandeen <sandeen@sandeen.net>
> 
> > ---
> >  db/type.c |    2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > 
> > diff --git a/db/type.c b/db/type.c
> > index f5f65042..d42935a2 100644
> > --- a/db/type.c
> > +++ b/db/type.c
> > @@ -107,7 +107,7 @@ static const typ_t	__typtab_crc[] = {

Heh.  Apparently the stgit rebase didn't notice that this patch was
already applied and cleverly went and broke non-sparse finobt.  And then
I didn't notice either. :( :(

Strike this patch, please.

--D

> >  	{ TYP_SYMLINK, "symlink", handle_struct, symlink_crc_hfld,
> >  		&xfs_symlink_buf_ops, XFS_SYMLINK_CRC_OFF },
> >  	{ TYP_TEXT, "text", handle_text, NULL, NULL, TYP_F_NO_CRC_OFF },
> > -	{ TYP_FINOBT, "finobt", handle_struct, inobt_crc_hfld,
> > +	{ TYP_FINOBT, "finobt", handle_struct, inobt_spcrc_hfld,
> >  		&xfs_inobt_buf_ops, XFS_BTREE_SBLOCK_CRC_OFF },
> >  	{ TYP_NONE, NULL }
> >  };
> > 

[PATCH 21/36] xfs_db: use TYP_FINOBT for finobt metadump

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Use the correct xfs_db type for dumping free inode btree blocks.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 db/metadump.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/db/metadump.c b/db/metadump.c
index 6ecd5685..3cce3012 100644
--- a/db/metadump.c
+++ b/db/metadump.c
@@ -2607,7 +2607,7 @@ copy_inodes(
 		levels = be32_to_cpu(agi->agi_free_level);
 
 		finobt = 1;
-		if (!scan_btree(agno, root, levels, TYP_INOBT, &finobt,
+		if (!scan_btree(agno, root, levels, TYP_FINOBT, &finobt,
 				scanfunc_ino))
 			return 0;
 	}

Re: [PATCH 21/36] xfs_db: use TYP_FINOBT for finobt metadump

[Eric Sandeen]

On 3/14/19 4:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Use the correct xfs_db type for dumping free inode btree blocks.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

wow this looks familiar too but ... this one isn't in yet :)

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  db/metadump.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> 
> diff --git a/db/metadump.c b/db/metadump.c
> index 6ecd5685..3cce3012 100644
> --- a/db/metadump.c
> +++ b/db/metadump.c
> @@ -2607,7 +2607,7 @@ copy_inodes(
>  		levels = be32_to_cpu(agi->agi_free_level);
>  
>  		finobt = 1;
> -		if (!scan_btree(agno, root, levels, TYP_INOBT, &finobt,
> +		if (!scan_btree(agno, root, levels, TYP_FINOBT, &finobt,
>  				scanfunc_ino))
>  			return 0;
>  	}
> 

[PATCH 22/36] xfs_info: use findmnt to handle mounted block devices

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Use findmnt to determine if the passed-in argument is associated with a
mount point, and if so, use spaceman to query the mounted filesystem.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 debian/control       |    2 +-
 spaceman/xfs_info.sh |   14 +++++++++++---
 2 files changed, 12 insertions(+), 4 deletions(-)


diff --git a/debian/control b/debian/control
index f4f807b0..0b3205f5 100644
--- a/debian/control
+++ b/debian/control
@@ -8,7 +8,7 @@ Standards-Version: 4.0.0
 Homepage: https://xfs.wiki.kernel.org/
 
 Package: xfsprogs
-Depends: ${shlibs:Depends}, ${misc:Depends}, python3:any
+Depends: ${shlibs:Depends}, ${misc:Depends}, python3:any, util-linux
 Provides: fsck-backend
 Suggests: xfsdump, acl, attr, quota
 Breaks: xfsdump (<< 3.0.0)
diff --git a/spaceman/xfs_info.sh b/spaceman/xfs_info.sh
index ecf17f61..70978164 100755
--- a/spaceman/xfs_info.sh
+++ b/spaceman/xfs_info.sh
@@ -24,11 +24,19 @@ set -- extra "$@"
 shift $OPTIND
 case $# in
 	1)
-		if [ -b "$1" ] || [ -f "$1" ]; then
-			xfs_db -p xfs_info -c "info" $OPTS "$1"
+		arg="$1"
+
+		# See if we can map the arg to a loop device
+		loopdev="$(losetup -n -O NAME -j "${arg}" 2> /dev/null)"
+		test -n "${loopdev}" && arg="${loopdev}"
+
+		# If we find a mountpoint for the device, do a live query;
+		# otherwise try reading the fs with xfs_db.
+		if mountpt="$(findmnt -f -n -o TARGET "${arg}" 2> /dev/null)"; then
+			xfs_spaceman -p xfs_info -c "info" $OPTS "${mountpt}"
 			status=$?
 		else
-			xfs_spaceman -p xfs_info -c "info" $OPTS "$1"
+			xfs_db -p xfs_info -c "info" $OPTS "${arg}"
 			status=$?
 		fi
 		;;

Re: [PATCH 22/36] xfs_info: use findmnt to handle mounted block devices

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Use findmnt to determine if the passed-in argument is associated with a
> mount point, and if so, use spaceman to query the mounted filesystem.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  debian/control       |    2 +-
>  spaceman/xfs_info.sh |   14 +++++++++++---
>  2 files changed, 12 insertions(+), 4 deletions(-)
> 
> 
> diff --git a/debian/control b/debian/control
> index f4f807b0..0b3205f5 100644
> --- a/debian/control
> +++ b/debian/control
> @@ -8,7 +8,7 @@ Standards-Version: 4.0.0
>  Homepage: https://xfs.wiki.kernel.org/
>  
>  Package: xfsprogs
> -Depends: ${shlibs:Depends}, ${misc:Depends}, python3:any
> +Depends: ${shlibs:Depends}, ${misc:Depends}, python3:any, util-linux
>  Provides: fsck-backend
>  Suggests: xfsdump, acl, attr, quota
>  Breaks: xfsdump (<< 3.0.0)
> diff --git a/spaceman/xfs_info.sh b/spaceman/xfs_info.sh
> index ecf17f61..70978164 100755
> --- a/spaceman/xfs_info.sh
> +++ b/spaceman/xfs_info.sh
> @@ -24,11 +24,19 @@ set -- extra "$@"
>  shift $OPTIND
>  case $# in
>  	1)
> -		if [ -b "$1" ] || [ -f "$1" ]; then
> -			xfs_db -p xfs_info -c "info" $OPTS "$1"
> +		arg="$1"
> +
> +		# See if we can map the arg to a loop device
> +		loopdev="$(losetup -n -O NAME -j "${arg}" 2> /dev/null)"

hm, -n has existed since 2013...

commit 9f56106df6b49864ba604f6824f9fad5aeabd17a
Author: Karel Zak <kzak@redhat.com>
Date:   Mon May 13 12:00:24 2013 +0200

    losetup: add --raw and --noheadings
    
    Signed-off-by: Karel Zak <kzak@redhat.com>

and -O since 2012:

commit 896352d3906c19e8519670f1a511afb01083ed5e
Author: Ondrej Oprala <ooprala@redhat.com>
Date:   Mon Nov 12 13:08:00 2012 +0100

    losetup: add --list and --output option

So, granted, that is a /long time/.  And yet, somehow, my RHEL7 box does not
have -n....

# losetup -j fsfile | awk -F: '{print $1}'
/dev/loop0

perhaps, for older compatibility?

(and do we need to handle association w/ more than 1 loop dev? urk...)


> +		test -n "${loopdev}" && arg="${loopdev}"
> +
> +		# If we find a mountpoint for the device, do a live query;
> +		# otherwise try reading the fs with xfs_db.
> +		if mountpt="$(findmnt -f -n -o TARGET "${arg}" 2> /dev/null)"; then
> +			xfs_spaceman -p xfs_info -c "info" $OPTS "${mountpt}"
>  			status=$?
>  		else
> -			xfs_spaceman -p xfs_info -c "info" $OPTS "$1"
> +			xfs_db -p xfs_info -c "info" $OPTS "${arg}"
>  			status=$?
>  		fi
>  		;;
> 

Re: [PATCH 22/36] xfs_info: use findmnt to handle mounted block devices

[Darrick J. Wong]

On Tue, Mar 26, 2019 at 12:28:43PM -0500, Eric Sandeen wrote:
> On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Use findmnt to determine if the passed-in argument is associated with a
> > mount point, and if so, use spaceman to query the mounted filesystem.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  debian/control       |    2 +-
> >  spaceman/xfs_info.sh |   14 +++++++++++---
> >  2 files changed, 12 insertions(+), 4 deletions(-)
> > 
> > 
> > diff --git a/debian/control b/debian/control
> > index f4f807b0..0b3205f5 100644
> > --- a/debian/control
> > +++ b/debian/control
> > @@ -8,7 +8,7 @@ Standards-Version: 4.0.0
> >  Homepage: https://xfs.wiki.kernel.org/
> >  
> >  Package: xfsprogs
> > -Depends: ${shlibs:Depends}, ${misc:Depends}, python3:any
> > +Depends: ${shlibs:Depends}, ${misc:Depends}, python3:any, util-linux
> >  Provides: fsck-backend
> >  Suggests: xfsdump, acl, attr, quota
> >  Breaks: xfsdump (<< 3.0.0)
> > diff --git a/spaceman/xfs_info.sh b/spaceman/xfs_info.sh
> > index ecf17f61..70978164 100755
> > --- a/spaceman/xfs_info.sh
> > +++ b/spaceman/xfs_info.sh
> > @@ -24,11 +24,19 @@ set -- extra "$@"
> >  shift $OPTIND
> >  case $# in
> >  	1)
> > -		if [ -b "$1" ] || [ -f "$1" ]; then
> > -			xfs_db -p xfs_info -c "info" $OPTS "$1"
> > +		arg="$1"
> > +
> > +		# See if we can map the arg to a loop device
> > +		loopdev="$(losetup -n -O NAME -j "${arg}" 2> /dev/null)"
> 
> hm, -n has existed since 2013...
> 
> commit 9f56106df6b49864ba604f6824f9fad5aeabd17a
> Author: Karel Zak <kzak@redhat.com>
> Date:   Mon May 13 12:00:24 2013 +0200
> 
>     losetup: add --raw and --noheadings
>     
>     Signed-off-by: Karel Zak <kzak@redhat.com>
> 
> and -O since 2012:
> 
> commit 896352d3906c19e8519670f1a511afb01083ed5e
> Author: Ondrej Oprala <ooprala@redhat.com>
> Date:   Mon Nov 12 13:08:00 2012 +0100
> 
>     losetup: add --list and --output option
> 
> So, granted, that is a /long time/.  And yet, somehow, my RHEL7 box does not
> have -n....
> 
> # losetup -j fsfile | awk -F: '{print $1}'
> /dev/loop0
> 
> perhaps, for older compatibility?
> 
> (and do we need to handle association w/ more than 1 loop dev? urk...)

Ok, I'll fix it to avoid losetup -n and only return the last one it
finds.

--D

> 
> 
> > +		test -n "${loopdev}" && arg="${loopdev}"
> > +
> > +		# If we find a mountpoint for the device, do a live query;
> > +		# otherwise try reading the fs with xfs_db.
> > +		if mountpt="$(findmnt -f -n -o TARGET "${arg}" 2> /dev/null)"; then
> > +			xfs_spaceman -p xfs_info -c "info" $OPTS "${mountpt}"
> >  			status=$?
> >  		else
> > -			xfs_spaceman -p xfs_info -c "info" $OPTS "$1"
> > +			xfs_db -p xfs_info -c "info" $OPTS "${arg}"
> >  			status=$?
> >  		fi
> >  		;;
> > 

[PATCH 23/36] xfs_repair: reinitialize the root directory nlink correctly

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

In mk_root_dir, we reinitialize the root directory inode with a link
count of 1.  This differs from mkfs parseproto, which initializes the
root to have a link count of 2.  The nlink discrepancy in repair is
caught and corrected during phase 7, but this is unnecessary since we
should set it properly in the first place.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 repair/phase6.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/repair/phase6.c b/repair/phase6.c
index 9477bc25..8a50b350 100644
--- a/repair/phase6.c
+++ b/repair/phase6.c
@@ -891,7 +891,7 @@ mk_root_dir(xfs_mount_t *mp)
 	ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
 	ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;
 
-	set_nlink(VFS_I(ip), 1);	/* account for . */
+	set_nlink(VFS_I(ip), 2);	/* account for . and .. */
 
 	times = XFS_ICHGTIME_CHG | XFS_ICHGTIME_MOD;
 	if (ip->i_d.di_version == 3) {

Re: [PATCH 23/36] xfs_repair: reinitialize the root directory nlink correctly

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> In mk_root_dir, we reinitialize the root directory inode with a link
> count of 1.  This differs from mkfs parseproto, which initializes the
> root to have a link count of 2.  The nlink discrepancy in repair is
> caught and corrected during phase 7, but this is unnecessary since we
> should set it properly in the first place.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

aaand review the most recent one.  o_O

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  repair/phase6.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> 
> diff --git a/repair/phase6.c b/repair/phase6.c
> index 9477bc25..8a50b350 100644
> --- a/repair/phase6.c
> +++ b/repair/phase6.c
> @@ -891,7 +891,7 @@ mk_root_dir(xfs_mount_t *mp)
>  	ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
>  	ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;
>  
> -	set_nlink(VFS_I(ip), 1);	/* account for . */
> +	set_nlink(VFS_I(ip), 2);	/* account for . and .. */
>  
>  	times = XFS_ICHGTIME_CHG | XFS_ICHGTIME_MOD;
>  	if (ip->i_d.di_version == 3) {
> 

[PATCH 24/36] xfs_repair: bump the irec on-disk nlink when adding lost+found

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

We increment the nlink of the root directory inode when creating a
"lost+found" directory during phase 6, but we don't update the irec copy
of the root dir nlink.  This normally gets papered over by phase 7, but
this can fail badly if:

1) The root directory had an entry to a busted subdirectory, so
   that root directory will have nlink == 3, but in the ino_tree,
   counted_nlinks == 2 and disk_nlinks == 3.

2) Phase 6 creates lost+found to root the files that were in the busted
   directory, we'll set nlink = 4 and counted_nlinks = 3.  The correct
   nlink is 3 ('.', '..', 'lost+found'), not 4.

3) During phase 7, we see that counted_nlinks == disk_nlinks and so we
   totally fail to correct the on-disk inode.

4) A subsequent run of xfs_repair complains about the nlink being 4
   instead of 3.

To fix this, we have to adjust the irec's disk_nlinks in step 2 so that
phase 7 seeds that counted_nlinks < disk_nlinks and resets nlink to
counted_nlinks.  This can be reproduced somewhat frequently by xfs/117.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 repair/phase6.c |   14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)


diff --git a/repair/phase6.c b/repair/phase6.c
index 8a50b350..194cfdbf 100644
--- a/repair/phase6.c
+++ b/repair/phase6.c
@@ -1019,6 +1019,7 @@ mk_orphanage(xfs_mount_t *mp)
 	 */
 	set_inode_used(irec, ino_offset);
 	add_inode_ref(irec, ino_offset);
+	add_inode_reached(irec, ino_offset);
 
 	/*
 	 * now that we know the transaction will stay around,
@@ -1037,14 +1038,14 @@ mk_orphanage(xfs_mount_t *mp)
 
 	/*
 	 * bump up the link count in the root directory to account
-	 * for .. in the new directory
+	 * for .. in the new directory, and update the irec copy of the
+	 * on-disk nlink so we don't fail the link count check later.
 	 */
 	inc_nlink(VFS_I(pip));
-	add_inode_ref(find_inode_rec(mp,
-				XFS_INO_TO_AGNO(mp, mp->m_sb.sb_rootino),
-				XFS_INO_TO_AGINO(mp, mp->m_sb.sb_rootino)), 0);
-
-
+	irec = find_inode_rec(mp, XFS_INO_TO_AGNO(mp, mp->m_sb.sb_rootino),
+				  XFS_INO_TO_AGINO(mp, mp->m_sb.sb_rootino));
+	add_inode_ref(irec, 0);
+	set_inode_disk_nlinks(irec, 0, get_inode_disk_nlinks(irec, 0) + 1);
 
 	libxfs_trans_log_inode(tp, pip, XFS_ILOG_CORE);
 	libxfs_dir_init(tp, ip, pip);
@@ -1056,7 +1057,6 @@ mk_orphanage(xfs_mount_t *mp)
 	}
 	libxfs_irele(ip);
 	libxfs_irele(pip);
-	add_inode_reached(irec,ino_offset);
 
 	return(ino);
 }

Re: [PATCH 24/36] xfs_repair: bump the irec on-disk nlink when adding lost+found

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> We increment the nlink of the root directory inode when creating a
> "lost+found" directory during phase 6, but we don't update the irec copy
> of the root dir nlink.  This normally gets papered over by phase 7, but
> this can fail badly if:
> 
> 1) The root directory had an entry to a busted subdirectory, so
>    that root directory will have nlink == 3, but in the ino_tree,
>    counted_nlinks == 2 and disk_nlinks == 3.
> 
> 2) Phase 6 creates lost+found to root the files that were in the busted
>    directory, we'll set nlink = 4 and counted_nlinks = 3.  The correct
>    nlink is 3 ('.', '..', 'lost+found'), not 4.
> 
> 3) During phase 7, we see that counted_nlinks == disk_nlinks and so we
>    totally fail to correct the on-disk inode.
> 
> 4) A subsequent run of xfs_repair complains about the nlink being 4
>    instead of 3.
> 
> To fix this, we have to adjust the irec's disk_nlinks in step 2 so that
> phase 7 seeds that counted_nlinks < disk_nlinks and resets nlink to
> counted_nlinks.  This can be reproduced somewhat frequently by xfs/117.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  repair/phase6.c |   14 +++++++-------
>  1 file changed, 7 insertions(+), 7 deletions(-)
> 
> 
> diff --git a/repair/phase6.c b/repair/phase6.c
> index 8a50b350..194cfdbf 100644
> --- a/repair/phase6.c
> +++ b/repair/phase6.c
> @@ -1019,6 +1019,7 @@ mk_orphanage(xfs_mount_t *mp)
>  	 */
>  	set_inode_used(irec, ino_offset);
>  	add_inode_ref(irec, ino_offset);
> +	add_inode_reached(irec, ino_offset);
>  
>  	/*
>  	 * now that we know the transaction will stay around,
> @@ -1037,14 +1038,14 @@ mk_orphanage(xfs_mount_t *mp)
>  
>  	/*
>  	 * bump up the link count in the root directory to account
> -	 * for .. in the new directory
> +	 * for .. in the new directory, and update the irec copy of the
> +	 * on-disk nlink so we don't fail the link count check later.
>  	 */
>  	inc_nlink(VFS_I(pip));
> -	add_inode_ref(find_inode_rec(mp,
> -				XFS_INO_TO_AGNO(mp, mp->m_sb.sb_rootino),
> -				XFS_INO_TO_AGINO(mp, mp->m_sb.sb_rootino)), 0);
> -
> -
> +	irec = find_inode_rec(mp, XFS_INO_TO_AGNO(mp, mp->m_sb.sb_rootino),
> +				  XFS_INO_TO_AGINO(mp, mp->m_sb.sb_rootino));
> +	add_inode_ref(irec, 0);
> +	set_inode_disk_nlinks(irec, 0, get_inode_disk_nlinks(irec, 0) + 1);
>  
>  	libxfs_trans_log_inode(tp, pip, XFS_ILOG_CORE);
>  	libxfs_dir_init(tp, ip, pip);
> @@ -1056,7 +1057,6 @@ mk_orphanage(xfs_mount_t *mp)
>  	}
>  	libxfs_irele(ip);
>  	libxfs_irele(pip);
> -	add_inode_reached(irec,ino_offset);
>  
>  	return(ino);
>  }
> 

[PATCH 25/36] xfs_repair: fix uninitialized variable warnings

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix some uninitialized variable warnings because ASSERT disappears if
DEBUG is not defined.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 repair/dinode.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


diff --git a/repair/dinode.c b/repair/dinode.c
index f670bf87..c0a56daa 100644
--- a/repair/dinode.c
+++ b/repair/dinode.c
@@ -1176,8 +1176,8 @@ process_quota_inode(
 	struct xfs_buf		*bp;
 	xfs_filblks_t		dqchunklen;
 	uint			dqperchunk;
-	int			quota_type;
-	char			*quota_string;
+	int			quota_type = 0;
+	char			*quota_string = NULL;
 	xfs_dqid_t		dqid;
 	xfs_fileoff_t		qbno;
 	int			i;

Re: [PATCH 25/36] xfs_repair: fix uninitialized variable warnings

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fix some uninitialized variable warnings because ASSERT disappears if
> DEBUG is not defined.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Fixes: commit 5857dce9e ("xfs_repair: check and repair quota metadata")

which was mine ;)

But, I wonder if we should really just happily carry on w/ an invalid
quota type 0 on a non-debug build.  I don't know how that will end.

Maybe we should just switch it to a hard assert(0) because it really
can only happen on a programming error today.


> ---
>  repair/dinode.c |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/repair/dinode.c b/repair/dinode.c
> index f670bf87..c0a56daa 100644
> --- a/repair/dinode.c
> +++ b/repair/dinode.c
> @@ -1176,8 +1176,8 @@ process_quota_inode(
>  	struct xfs_buf		*bp;
>  	xfs_filblks_t		dqchunklen;
>  	uint			dqperchunk;
> -	int			quota_type;
> -	char			*quota_string;
> +	int			quota_type = 0;
> +	char			*quota_string = NULL;
>  	xfs_dqid_t		dqid;
>  	xfs_fileoff_t		qbno;
>  	int			i;
> 

Re: [PATCH 25/36] xfs_repair: fix uninitialized variable warnings

[Darrick J. Wong]

On Tue, Mar 26, 2019 at 02:56:02PM -0500, Eric Sandeen wrote:
> On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Fix some uninitialized variable warnings because ASSERT disappears if
> > DEBUG is not defined.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fixes: commit 5857dce9e ("xfs_repair: check and repair quota metadata")
> 
> which was mine ;)
> 
> But, I wonder if we should really just happily carry on w/ an invalid
> quota type 0 on a non-debug build.  I don't know how that will end.
> 
> Maybe we should just switch it to a hard assert(0) because it really
> can only happen on a programming error today.

Yeah, we ought to be more defensive about that, since in theory we could
introduce a subtle bug some day where we validate a corrupt dquot with
flags == 0.

/me kinda wonders why we define away the asserts for repair, since it's
not like continuing under invalid premises in a repair tool is a good
idea.

--D

> 
> > ---
> >  repair/dinode.c |    4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > 
> > diff --git a/repair/dinode.c b/repair/dinode.c
> > index f670bf87..c0a56daa 100644
> > --- a/repair/dinode.c
> > +++ b/repair/dinode.c
> > @@ -1176,8 +1176,8 @@ process_quota_inode(
> >  	struct xfs_buf		*bp;
> >  	xfs_filblks_t		dqchunklen;
> >  	uint			dqperchunk;
> > -	int			quota_type;
> > -	char			*quota_string;
> > +	int			quota_type = 0;
> > +	char			*quota_string = NULL;
> >  	xfs_dqid_t		dqid;
> >  	xfs_fileoff_t		qbno;
> >  	int			i;
> > 

Re: [PATCH 25/36] xfs_repair: fix uninitialized variable warnings

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fix some uninitialized variable warnings because ASSERT disappears if
> DEBUG is not defined.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  repair/dinode.c |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/repair/dinode.c b/repair/dinode.c
> index f670bf87..c0a56daa 100644
> --- a/repair/dinode.c
> +++ b/repair/dinode.c
> @@ -1176,8 +1176,8 @@ process_quota_inode(
>  	struct xfs_buf		*bp;
>  	xfs_filblks_t		dqchunklen;
>  	uint			dqperchunk;
> -	int			quota_type;
> -	char			*quota_string;
> +	int			quota_type = 0;
> +	char			*quota_string = NULL;
>  	xfs_dqid_t		dqid;
>  	xfs_fileoff_t		qbno;
>  	int			i;
> 

well, I was concerned that if we get here with an invalid inode type,
we'll use an invalid quota type, and <unknown> will happen.  But
we're relying on the case statement which calls us, which has only
valid types. ... I guess we could s/ASSERT/assert/ so it's there,
or just don't worry be happy.  There doesn't seem to be much
appetite for spot-fixing this, so

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

[PATCH 26/36] xfs_repair: refactor buffer ops assignments during phase 5

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Refactor the buffer ops assignments in phase 5 to use a helper function
to determine the correct buf_ops instead of open-coding them.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 repair/phase5.c |   34 ++++++++++++++++++++++++++++------
 1 file changed, 28 insertions(+), 6 deletions(-)


diff --git a/repair/phase5.c b/repair/phase5.c
index 367c3ab9..0c53096f 100644
--- a/repair/phase5.c
+++ b/repair/phase5.c
@@ -615,6 +615,24 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	return(extra_blocks);
 }
 
+/* Map btnum to buffer ops for the types that need it. */
+static const struct xfs_buf_ops *
+btnum_to_ops(
+	xfs_btnum_t	btnum)
+{
+	switch (btnum) {
+	case XFS_BTNUM_BNO:
+	case XFS_BTNUM_CNT:
+		return &xfs_allocbt_buf_ops;
+	case XFS_BTNUM_INO:
+	case XFS_BTNUM_FINO:
+		return &xfs_inobt_buf_ops;
+	default:
+		ASSERT(0);
+		return NULL;
+	}
+}
+
 static void
 prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 		bt_status_t *btree_curs, xfs_agblock_t startblock,
@@ -625,6 +643,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	xfs_alloc_ptr_t		*bt_ptr;
 	xfs_agblock_t		agbno;
 	bt_stat_level_t		*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
 
 	ASSERT(btnum == XFS_BTNUM_BNO || btnum == XFS_BTNUM_CNT);
 
@@ -675,7 +694,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 		/*
 		 * initialize block header
 		 */
-		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, level,
@@ -723,6 +742,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 	extent_tree_node_t	*ext_ptr;
 	bt_stat_level_t		*lptr;
 	xfs_extlen_t		freeblks;
+	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
 
 	ASSERT(btnum == XFS_BTNUM_BNO || btnum == XFS_BTNUM_CNT);
 
@@ -754,7 +774,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 		/*
 		 * initialize block header
 		 */
-		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, i, 0, agno, 0);
@@ -781,7 +801,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 		/*
 		 * block initialization, lay in block header
 		 */
-		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, 0, 0, agno, 0);
@@ -990,6 +1010,7 @@ prop_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs,
 	xfs_inobt_ptr_t		*bt_ptr;
 	xfs_agblock_t		agbno;
 	bt_stat_level_t		*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
 
 	level++;
 
@@ -1038,7 +1059,7 @@ prop_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs,
 		/*
 		 * initialize block header
 		 */
-		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum,
@@ -1130,6 +1151,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 	xfs_inobt_rec_t		*bt_rec;
 	ino_tree_node_t		*ino_rec;
 	bt_stat_level_t		*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
 	xfs_agino_t		count = 0;
 	xfs_agino_t		freecount = 0;
 	int			inocnt;
@@ -1160,7 +1182,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 		 * initialize block header
 		 */
 
-		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, i, 0, agno, 0);
@@ -1188,7 +1210,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 		/*
 		 * block initialization, lay in block header
 		 */
-		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, 0, 0, agno, 0);

Re: [PATCH 26/36] xfs_repair: refactor buffer ops assignments during phase 5

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Refactor the buffer ops assignments in phase 5 to use a helper function
> to determine the correct buf_ops instead of open-coding them.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  repair/phase5.c |   34 ++++++++++++++++++++++++++++------
>  1 file changed, 28 insertions(+), 6 deletions(-)

This is bugging me a bit from a symmetry POV in terms of when your
helper is used and when it's not... a generic helper that only
helps a subset of the btnums seems weird to me.

(I get it that in this case it's because 2 types use the same
ops, but still)

C symbol: b_ops

  File                 Function                 Line
0 repair/dino_chunks.c check_aginode_block        57 bp->b_ops = &xfs_inode_buf_ops;
1 repair/dino_chunks.c process_inode_chunk	 682 bplist[bp_index]->b_ops = &xfs_inode_buf_ops;
2 repair/phase5.c      prop_freespace_cursor     697 lptr->buf_p->b_ops = ops;
3 repair/phase5.c      build_freespace_tree	 777 lptr->buf_p->b_ops = ops;
4 repair/phase5.c      build_freespace_tree	 804 lptr->buf_p->b_ops = ops;
5 repair/phase5.c      prop_ino_cursor          1062 lptr->buf_p->b_ops = ops;
6 repair/phase5.c      build_agi                1104 agi_buf->b_ops = &xfs_agi_buf_ops;
7 repair/phase5.c      build_ino_tree           1185 lptr->buf_p->b_ops = ops;
8 repair/phase5.c      build_ino_tree           1213 lptr->buf_p->b_ops = ops;
9 repair/phase5.c      prop_rmap_cursor         1466 lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
a repair/phase5.c      build_rmap_tree          1578 lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
b repair/phase5.c      build_rmap_tree          1605 lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
c repair/phase5.c      prop_refc_cursor         1814 lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
d repair/phase5.c      build_refcount_tree	1881 lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
e repair/phase5.c      build_refcount_tree	1908 lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
f repair/phase5.c      build_agf_agfl           1992 agf_buf->b_ops = &xfs_agf_buf_ops;
g repair/phase5.c      build_agf_agfl           2064 agfl_buf->b_ops = &xfs_agfl_buf_ops;
h repair/phase6.c      dir_read_buf              205 (*bpp)->b_ops = ops;
i repair/phase6.c      longform_dir2_check_node 2181 bp->b_ops = &xfs_dir3_leafn_buf_ops;


> 
> diff --git a/repair/phase5.c b/repair/phase5.c
> index 367c3ab9..0c53096f 100644
> --- a/repair/phase5.c
> +++ b/repair/phase5.c
> @@ -615,6 +615,24 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
>  	return(extra_blocks);
>  }
>  
> +/* Map btnum to buffer ops for the types that need it. */
> +static const struct xfs_buf_ops *
> +btnum_to_ops(
> +	xfs_btnum_t	btnum)
> +{
> +	switch (btnum) {
> +	case XFS_BTNUM_BNO:
> +	case XFS_BTNUM_CNT:
> +		return &xfs_allocbt_buf_ops;
> +	case XFS_BTNUM_INO:
> +	case XFS_BTNUM_FINO:
> +		return &xfs_inobt_buf_ops;
> +	default:
> +		ASSERT(0);
> +		return NULL;
> +	}
> +}
> +
>  static void
>  prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		bt_status_t *btree_curs, xfs_agblock_t startblock,
> @@ -625,6 +643,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
>  	xfs_alloc_ptr_t		*bt_ptr;
>  	xfs_agblock_t		agbno;
>  	bt_stat_level_t		*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
>  
>  	ASSERT(btnum == XFS_BTNUM_BNO || btnum == XFS_BTNUM_CNT);
>  
> @@ -675,7 +694,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		/*
>  		 * initialize block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, level,
> @@ -723,6 +742,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  	extent_tree_node_t	*ext_ptr;
>  	bt_stat_level_t		*lptr;
>  	xfs_extlen_t		freeblks;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
>  
>  	ASSERT(btnum == XFS_BTNUM_BNO || btnum == XFS_BTNUM_CNT);
>  
> @@ -754,7 +774,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		/*
>  		 * initialize block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, i, 0, agno, 0);
> @@ -781,7 +801,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		/*
>  		 * block initialization, lay in block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, 0, 0, agno, 0);
> @@ -990,6 +1010,7 @@ prop_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs,
>  	xfs_inobt_ptr_t		*bt_ptr;
>  	xfs_agblock_t		agbno;
>  	bt_stat_level_t		*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
>  
>  	level++;
>  
> @@ -1038,7 +1059,7 @@ prop_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs,
>  		/*
>  		 * initialize block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum,
> @@ -1130,6 +1151,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  	xfs_inobt_rec_t		*bt_rec;
>  	ino_tree_node_t		*ino_rec;
>  	bt_stat_level_t		*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
>  	xfs_agino_t		count = 0;
>  	xfs_agino_t		freecount = 0;
>  	int			inocnt;
> @@ -1160,7 +1182,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		 * initialize block header
>  		 */
>  
> -		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, i, 0, agno, 0);
> @@ -1188,7 +1210,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		/*
>  		 * block initialization, lay in block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, 0, 0, agno, 0);
> 

[PATCH v2 26/36] xfs_repair: refactor buffer ops assignments during phase 5

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Refactor the buffer ops assignments in phase 5 to use a helper function
to determine the correct buf_ops instead of open-coding them.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
v2: convert all the ag btree types
---
 repair/phase5.c |   54 ++++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 42 insertions(+), 12 deletions(-)

diff --git a/repair/phase5.c b/repair/phase5.c
index 367c3ab9..5d9c542a 100644
--- a/repair/phase5.c
+++ b/repair/phase5.c
@@ -615,6 +615,28 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	return(extra_blocks);
 }
 
+/* Map btnum to buffer ops for the types that need it. */
+static const struct xfs_buf_ops *
+btnum_to_ops(
+	xfs_btnum_t	btnum)
+{
+	switch (btnum) {
+	case XFS_BTNUM_BNO:
+	case XFS_BTNUM_CNT:
+		return &xfs_allocbt_buf_ops;
+	case XFS_BTNUM_INO:
+	case XFS_BTNUM_FINO:
+		return &xfs_inobt_buf_ops;
+	case XFS_BTNUM_RMAP:
+		return &xfs_rmapbt_buf_ops;
+	case XFS_BTNUM_REFC:
+		return &xfs_refcountbt_buf_ops;
+	default:
+		ASSERT(0);
+		return NULL;
+	}
+}
+
 static void
 prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 		bt_status_t *btree_curs, xfs_agblock_t startblock,
@@ -625,6 +647,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 	xfs_alloc_ptr_t		*bt_ptr;
 	xfs_agblock_t		agbno;
 	bt_stat_level_t		*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
 
 	ASSERT(btnum == XFS_BTNUM_BNO || btnum == XFS_BTNUM_CNT);
 
@@ -675,7 +698,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
 		/*
 		 * initialize block header
 		 */
-		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, level,
@@ -723,6 +746,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 	extent_tree_node_t	*ext_ptr;
 	bt_stat_level_t		*lptr;
 	xfs_extlen_t		freeblks;
+	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
 
 	ASSERT(btnum == XFS_BTNUM_BNO || btnum == XFS_BTNUM_CNT);
 
@@ -754,7 +778,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 		/*
 		 * initialize block header
 		 */
-		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, i, 0, agno, 0);
@@ -781,7 +805,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 		/*
 		 * block initialization, lay in block header
 		 */
-		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, 0, 0, agno, 0);
@@ -990,6 +1014,7 @@ prop_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs,
 	xfs_inobt_ptr_t		*bt_ptr;
 	xfs_agblock_t		agbno;
 	bt_stat_level_t		*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
 
 	level++;
 
@@ -1038,7 +1063,7 @@ prop_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs,
 		/*
 		 * initialize block header
 		 */
-		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum,
@@ -1130,6 +1155,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 	xfs_inobt_rec_t		*bt_rec;
 	ino_tree_node_t		*ino_rec;
 	bt_stat_level_t		*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
 	xfs_agino_t		count = 0;
 	xfs_agino_t		freecount = 0;
 	int			inocnt;
@@ -1160,7 +1186,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 		 * initialize block header
 		 */
 
-		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, i, 0, agno, 0);
@@ -1188,7 +1214,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
 		/*
 		 * block initialization, lay in block header
 		 */
-		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, btnum, 0, 0, agno, 0);
@@ -1393,6 +1419,7 @@ prop_rmap_cursor(
 	xfs_rmap_ptr_t		*bt_ptr;
 	xfs_agblock_t		agbno;
 	struct bt_stat_level	*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(XFS_BTNUM_RMAP);
 
 	level++;
 
@@ -1441,7 +1468,7 @@ prop_rmap_cursor(
 		/*
 		 * initialize block header
 		 */
-		lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_RMAP,
@@ -1530,6 +1557,7 @@ build_rmap_tree(
 	struct xfs_rmap_irec	highest_key = {0};
 	struct xfs_rmap_irec	hi_key = {0};
 	struct bt_stat_level	*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(XFS_BTNUM_RMAP);
 	int			numrecs;
 	int			level = btree_curs->num_levels;
 	int			error;
@@ -1553,7 +1581,7 @@ build_rmap_tree(
 		 * initialize block header
 		 */
 
-		lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_RMAP,
@@ -1580,7 +1608,7 @@ _("Insufficient memory to construct reverse-map cursor."));
 		/*
 		 * block initialization, lay in block header
 		 */
-		lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_RMAP,
@@ -1741,6 +1769,7 @@ prop_refc_cursor(
 	xfs_refcount_ptr_t	*bt_ptr;
 	xfs_agblock_t		agbno;
 	struct bt_stat_level	*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(XFS_BTNUM_REFC);
 
 	level++;
 
@@ -1789,7 +1818,7 @@ prop_refc_cursor(
 		/*
 		 * initialize block header
 		 */
-		lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_REFC,
@@ -1834,6 +1863,7 @@ build_refcount_tree(
 	struct xfs_slab_cursor	*refc_cur;
 	struct xfs_refcount_rec	*bt_rec;
 	struct bt_stat_level	*lptr;
+	const struct xfs_buf_ops *ops = btnum_to_ops(XFS_BTNUM_REFC);
 	int			numrecs;
 	int			level = btree_curs->num_levels;
 	int			error;
@@ -1856,7 +1886,7 @@ build_refcount_tree(
 		 * initialize block header
 		 */
 
-		lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_REFC,
@@ -1883,7 +1913,7 @@ _("Insufficient memory to construct refcount cursor."));
 		/*
 		 * block initialization, lay in block header
 		 */
-		lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
+		lptr->buf_p->b_ops = ops;
 		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
 		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
 		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_REFC,

Re: [PATCH v2 26/36] xfs_repair: refactor buffer ops assignments during phase 5

[Eric Sandeen]

On 4/10/19 10:25 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Refactor the buffer ops assignments in phase 5 to use a helper function
> to determine the correct buf_ops instead of open-coding them.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
> v2: convert all the ag btree types

yay for consistency, thanks.

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  repair/phase5.c |   54 ++++++++++++++++++++++++++++++++++++++++++------------
>  1 file changed, 42 insertions(+), 12 deletions(-)
> 
> diff --git a/repair/phase5.c b/repair/phase5.c
> index 367c3ab9..5d9c542a 100644
> --- a/repair/phase5.c
> +++ b/repair/phase5.c
> @@ -615,6 +615,28 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
>  	return(extra_blocks);
>  }
>  
> +/* Map btnum to buffer ops for the types that need it. */
> +static const struct xfs_buf_ops *
> +btnum_to_ops(
> +	xfs_btnum_t	btnum)
> +{
> +	switch (btnum) {
> +	case XFS_BTNUM_BNO:
> +	case XFS_BTNUM_CNT:
> +		return &xfs_allocbt_buf_ops;
> +	case XFS_BTNUM_INO:
> +	case XFS_BTNUM_FINO:
> +		return &xfs_inobt_buf_ops;
> +	case XFS_BTNUM_RMAP:
> +		return &xfs_rmapbt_buf_ops;
> +	case XFS_BTNUM_REFC:
> +		return &xfs_refcountbt_buf_ops;
> +	default:
> +		ASSERT(0);
> +		return NULL;
> +	}
> +}
> +
>  static void
>  prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		bt_status_t *btree_curs, xfs_agblock_t startblock,
> @@ -625,6 +647,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
>  	xfs_alloc_ptr_t		*bt_ptr;
>  	xfs_agblock_t		agbno;
>  	bt_stat_level_t		*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
>  
>  	ASSERT(btnum == XFS_BTNUM_BNO || btnum == XFS_BTNUM_CNT);
>  
> @@ -675,7 +698,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		/*
>  		 * initialize block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, level,
> @@ -723,6 +746,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  	extent_tree_node_t	*ext_ptr;
>  	bt_stat_level_t		*lptr;
>  	xfs_extlen_t		freeblks;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
>  
>  	ASSERT(btnum == XFS_BTNUM_BNO || btnum == XFS_BTNUM_CNT);
>  
> @@ -754,7 +778,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		/*
>  		 * initialize block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, i, 0, agno, 0);
> @@ -781,7 +805,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		/*
>  		 * block initialization, lay in block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_allocbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, 0, 0, agno, 0);
> @@ -990,6 +1014,7 @@ prop_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs,
>  	xfs_inobt_ptr_t		*bt_ptr;
>  	xfs_agblock_t		agbno;
>  	bt_stat_level_t		*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
>  
>  	level++;
>  
> @@ -1038,7 +1063,7 @@ prop_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs,
>  		/*
>  		 * initialize block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum,
> @@ -1130,6 +1155,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  	xfs_inobt_rec_t		*bt_rec;
>  	ino_tree_node_t		*ino_rec;
>  	bt_stat_level_t		*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(btnum);
>  	xfs_agino_t		count = 0;
>  	xfs_agino_t		freecount = 0;
>  	int			inocnt;
> @@ -1160,7 +1186,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		 * initialize block header
>  		 */
>  
> -		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, i, 0, agno, 0);
> @@ -1188,7 +1214,7 @@ build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno,
>  		/*
>  		 * block initialization, lay in block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_inobt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, btnum, 0, 0, agno, 0);
> @@ -1393,6 +1419,7 @@ prop_rmap_cursor(
>  	xfs_rmap_ptr_t		*bt_ptr;
>  	xfs_agblock_t		agbno;
>  	struct bt_stat_level	*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(XFS_BTNUM_RMAP);
>  
>  	level++;
>  
> @@ -1441,7 +1468,7 @@ prop_rmap_cursor(
>  		/*
>  		 * initialize block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_RMAP,
> @@ -1530,6 +1557,7 @@ build_rmap_tree(
>  	struct xfs_rmap_irec	highest_key = {0};
>  	struct xfs_rmap_irec	hi_key = {0};
>  	struct bt_stat_level	*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(XFS_BTNUM_RMAP);
>  	int			numrecs;
>  	int			level = btree_curs->num_levels;
>  	int			error;
> @@ -1553,7 +1581,7 @@ build_rmap_tree(
>  		 * initialize block header
>  		 */
>  
> -		lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_RMAP,
> @@ -1580,7 +1608,7 @@ _("Insufficient memory to construct reverse-map cursor."));
>  		/*
>  		 * block initialization, lay in block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_rmapbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_RMAP,
> @@ -1741,6 +1769,7 @@ prop_refc_cursor(
>  	xfs_refcount_ptr_t	*bt_ptr;
>  	xfs_agblock_t		agbno;
>  	struct bt_stat_level	*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(XFS_BTNUM_REFC);
>  
>  	level++;
>  
> @@ -1789,7 +1818,7 @@ prop_refc_cursor(
>  		/*
>  		 * initialize block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_REFC,
> @@ -1834,6 +1863,7 @@ build_refcount_tree(
>  	struct xfs_slab_cursor	*refc_cur;
>  	struct xfs_refcount_rec	*bt_rec;
>  	struct bt_stat_level	*lptr;
> +	const struct xfs_buf_ops *ops = btnum_to_ops(XFS_BTNUM_REFC);
>  	int			numrecs;
>  	int			level = btree_curs->num_levels;
>  	int			error;
> @@ -1856,7 +1886,7 @@ build_refcount_tree(
>  		 * initialize block header
>  		 */
>  
> -		lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_REFC,
> @@ -1883,7 +1913,7 @@ _("Insufficient memory to construct refcount cursor."));
>  		/*
>  		 * block initialization, lay in block header
>  		 */
> -		lptr->buf_p->b_ops = &xfs_refcountbt_buf_ops;
> +		lptr->buf_p->b_ops = ops;
>  		bt_hdr = XFS_BUF_TO_BLOCK(lptr->buf_p);
>  		memset(bt_hdr, 0, mp->m_sb.sb_blocksize);
>  		libxfs_btree_init_block(mp, lptr->buf_p, XFS_BTNUM_REFC,
> 

[PATCH 27/36] xfs_repair: pass ops through during scan

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Pass the buffer ops through scan_sbtree so that we detect finobt blocks
properly and we don't have to keep switching on magic numbers for the
free space btrees.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 repair/scan.c |   39 ++++++++++++++++-----------------------
 1 file changed, 16 insertions(+), 23 deletions(-)


diff --git a/repair/scan.c b/repair/scan.c
index 12ca3144..edc375a9 100644
--- a/repair/scan.c
+++ b/repair/scan.c
@@ -59,7 +59,8 @@ scan_sbtree(
 				int			suspect,
 				int			isroot,
 				uint32_t		magic,
-				void			*priv),
+				void			*priv,
+				const struct xfs_buf_ops *ops),
 	int		isroot,
 	uint32_t	magic,
 	void		*priv,
@@ -80,7 +81,7 @@ scan_sbtree(
 	}
 
 	(*func)(XFS_BUF_TO_BLOCK(bp), nlevels - 1, root, agno, suspect,
-							isroot, magic, priv);
+			isroot, magic, priv, ops);
 	libxfs_putbuf(bp);
 }
 
@@ -543,7 +544,8 @@ scan_allocbt(
 	int			suspect,
 	int			isroot,
 	uint32_t		magic,
-	void			*priv)
+	void			*priv,
+	const struct xfs_buf_ops *ops)
 {
 	struct aghdr_cnts	*agcnts = priv;
 	const char 		*name;
@@ -750,20 +752,8 @@ _("%s freespace btree block claimed (state %d), agno %d, bno %d, suspect %d\n"),
 		 * as possible.
 		 */
 		if (agbno != 0 && verify_agbno(mp, agno, agbno)) {
-			switch (magic) {
-			case XFS_ABTB_CRC_MAGIC:
-			case XFS_ABTB_MAGIC:
-				scan_sbtree(agbno, level, agno, suspect,
-					    scan_allocbt, 0, magic, priv,
-					    &xfs_allocbt_buf_ops);
-				break;
-			case XFS_ABTC_CRC_MAGIC:
-			case XFS_ABTC_MAGIC:
-				scan_sbtree(agbno, level, agno, suspect,
-					    scan_allocbt, 0, magic, priv,
-					    &xfs_allocbt_buf_ops);
-				break;
-			}
+			scan_sbtree(agbno, level, agno, suspect, scan_allocbt,
+				    0, magic, priv, ops);
 		}
 	}
 }
@@ -926,7 +916,8 @@ scan_rmapbt(
 	int			suspect,
 	int			isroot,
 	uint32_t		magic,
-	void			*priv)
+	void			*priv,
+	const struct xfs_buf_ops *ops)
 {
 	const char		*name = "rmap";
 	int			i;
@@ -1205,7 +1196,7 @@ _("%s rmap btree block claimed (state %d), agno %d, bno %d, suspect %d\n"),
 
 		if (agbno != 0 && verify_agbno(mp, agno, agbno)) {
 			scan_sbtree(agbno, level, agno, suspect, scan_rmapbt, 0,
-				    magic, priv, &xfs_rmapbt_buf_ops);
+				    magic, priv, ops);
 		}
 	}
 
@@ -1229,7 +1220,8 @@ scan_refcbt(
 	int			suspect,
 	int			isroot,
 	uint32_t		magic,
-	void			*priv)
+	void			*priv,
+	const struct xfs_buf_ops *ops)
 {
 	const char		*name = "refcount";
 	int			i;
@@ -1423,7 +1415,7 @@ _("extent (%u/%u) len %u claimed, state is %d\n"),
 
 		if (agbno != 0 && verify_agbno(mp, agno, agbno)) {
 			scan_sbtree(agbno, level, agno, suspect, scan_refcbt, 0,
-				    magic, priv, &xfs_refcountbt_buf_ops);
+				    magic, priv, ops);
 		}
 	}
 out:
@@ -1935,7 +1927,8 @@ scan_inobt(
 	int			suspect,
 	int			isroot,
 	uint32_t		magic,
-	void			*priv)
+	void			*priv,
+	const struct xfs_buf_ops *ops)
 {
 	struct aghdr_cnts	*agcnts = priv;
 	int			i;
@@ -2091,7 +2084,7 @@ _("inode btree block claimed (state %d), agno %d, bno %d, suspect %d\n"),
 							be32_to_cpu(pp[i])))
 			scan_sbtree(be32_to_cpu(pp[i]), level, agno,
 					suspect, scan_inobt, 0, magic, priv,
-					&xfs_inobt_buf_ops);
+					ops);
 	}
 }
 

Re: [PATCH 27/36] xfs_repair: pass ops through during scan

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Pass the buffer ops through scan_sbtree so that we detect finobt blocks
> properly and we don't have to keep switching on magic numbers for the
> free space btrees.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  repair/scan.c |   39 ++++++++++++++++-----------------------
>  1 file changed, 16 insertions(+), 23 deletions(-)
> 
> 
> diff --git a/repair/scan.c b/repair/scan.c
> index 12ca3144..edc375a9 100644
> --- a/repair/scan.c
> +++ b/repair/scan.c
> @@ -59,7 +59,8 @@ scan_sbtree(
>  				int			suspect,
>  				int			isroot,
>  				uint32_t		magic,
> -				void			*priv),
> +				void			*priv,
> +				const struct xfs_buf_ops *ops),
>  	int		isroot,
>  	uint32_t	magic,
>  	void		*priv,
> @@ -80,7 +81,7 @@ scan_sbtree(
>  	}
>  
>  	(*func)(XFS_BUF_TO_BLOCK(bp), nlevels - 1, root, agno, suspect,
> -							isroot, magic, priv);
> +			isroot, magic, priv, ops);
>  	libxfs_putbuf(bp);
>  }
>  
> @@ -543,7 +544,8 @@ scan_allocbt(
>  	int			suspect,
>  	int			isroot,
>  	uint32_t		magic,
> -	void			*priv)
> +	void			*priv,
> +	const struct xfs_buf_ops *ops)
>  {
>  	struct aghdr_cnts	*agcnts = priv;
>  	const char 		*name;
> @@ -750,20 +752,8 @@ _("%s freespace btree block claimed (state %d), agno %d, bno %d, suspect %d\n"),
>  		 * as possible.
>  		 */
>  		if (agbno != 0 && verify_agbno(mp, agno, agbno)) {
> -			switch (magic) {
> -			case XFS_ABTB_CRC_MAGIC:
> -			case XFS_ABTB_MAGIC:
> -				scan_sbtree(agbno, level, agno, suspect,
> -					    scan_allocbt, 0, magic, priv,
> -					    &xfs_allocbt_buf_ops);
> -				break;
> -			case XFS_ABTC_CRC_MAGIC:
> -			case XFS_ABTC_MAGIC:
> -				scan_sbtree(agbno, level, agno, suspect,
> -					    scan_allocbt, 0, magic, priv,
> -					    &xfs_allocbt_buf_ops);
> -				break;
> -			}
> +			scan_sbtree(agbno, level, agno, suspect, scan_allocbt,
> +				    0, magic, priv, ops);
>  		}
>  	}
>  }
> @@ -926,7 +916,8 @@ scan_rmapbt(
>  	int			suspect,
>  	int			isroot,
>  	uint32_t		magic,
> -	void			*priv)
> +	void			*priv,
> +	const struct xfs_buf_ops *ops)
>  {
>  	const char		*name = "rmap";
>  	int			i;
> @@ -1205,7 +1196,7 @@ _("%s rmap btree block claimed (state %d), agno %d, bno %d, suspect %d\n"),
>  
>  		if (agbno != 0 && verify_agbno(mp, agno, agbno)) {
>  			scan_sbtree(agbno, level, agno, suspect, scan_rmapbt, 0,
> -				    magic, priv, &xfs_rmapbt_buf_ops);
> +				    magic, priv, ops);
>  		}
>  	}
>  
> @@ -1229,7 +1220,8 @@ scan_refcbt(
>  	int			suspect,
>  	int			isroot,
>  	uint32_t		magic,
> -	void			*priv)
> +	void			*priv,
> +	const struct xfs_buf_ops *ops)
>  {
>  	const char		*name = "refcount";
>  	int			i;
> @@ -1423,7 +1415,7 @@ _("extent (%u/%u) len %u claimed, state is %d\n"),
>  
>  		if (agbno != 0 && verify_agbno(mp, agno, agbno)) {
>  			scan_sbtree(agbno, level, agno, suspect, scan_refcbt, 0,
> -				    magic, priv, &xfs_refcountbt_buf_ops);
> +				    magic, priv, ops);
>  		}
>  	}
>  out:
> @@ -1935,7 +1927,8 @@ scan_inobt(
>  	int			suspect,
>  	int			isroot,
>  	uint32_t		magic,
> -	void			*priv)
> +	void			*priv,
> +	const struct xfs_buf_ops *ops)
>  {
>  	struct aghdr_cnts	*agcnts = priv;
>  	int			i;
> @@ -2091,7 +2084,7 @@ _("inode btree block claimed (state %d), agno %d, bno %d, suspect %d\n"),
>  							be32_to_cpu(pp[i])))
>  			scan_sbtree(be32_to_cpu(pp[i]), level, agno,
>  					suspect, scan_inobt, 0, magic, priv,
> -					&xfs_inobt_buf_ops);
> +					ops);
>  	}
>  }
>  
> 

[PATCH 28/36] libfrog: hoist bitmap out of scrub

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Move the bitmap code to libfrog so that we can use bitmaps in
xfs_repair.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 include/bitmap.h |   24 +++
 libfrog/Makefile |    1 
 libfrog/bitmap.c |  393 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 scrub/Makefile   |    2 
 scrub/bitmap.c   |  393 ------------------------------------------------------
 scrub/bitmap.h   |   24 ---
 6 files changed, 418 insertions(+), 419 deletions(-)
 create mode 100644 include/bitmap.h
 create mode 100644 libfrog/bitmap.c
 delete mode 100644 scrub/bitmap.c
 delete mode 100644 scrub/bitmap.h


diff --git a/include/bitmap.h b/include/bitmap.h
new file mode 100644
index 00000000..e29a4335
--- /dev/null
+++ b/include/bitmap.h
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2018 Oracle.  All Rights Reserved.
+ * Author: Darrick J. Wong <darrick.wong@oracle.com>
+ */
+#ifndef LIBFROG_BITMAP_H_
+#define LIBFROG_BITMAP_H_
+
+struct bitmap {
+	pthread_mutex_t		bt_lock;
+	struct avl64tree_desc	*bt_tree;
+};
+
+bool bitmap_init(struct bitmap **bmap);
+void bitmap_free(struct bitmap **bmap);
+bool bitmap_set(struct bitmap *bmap, uint64_t start, uint64_t length);
+bool bitmap_iterate(struct bitmap *bmap,
+		bool (*fn)(uint64_t, uint64_t, void *), void *arg);
+bool bitmap_test(struct bitmap *bmap, uint64_t start,
+		uint64_t len);
+bool bitmap_empty(struct bitmap *bmap);
+void bitmap_dump(struct bitmap *bmap);
+
+#endif /* LIBFROG_BITMAP_H_ */
diff --git a/libfrog/Makefile b/libfrog/Makefile
index dbff9596..f5a0539b 100644
--- a/libfrog/Makefile
+++ b/libfrog/Makefile
@@ -12,6 +12,7 @@ LT_AGE = 0
 
 CFILES = \
 avl64.c \
+bitmap.c \
 convert.c \
 crc32.c \
 fsgeom.c \
diff --git a/libfrog/bitmap.c b/libfrog/bitmap.c
new file mode 100644
index 00000000..aecdba0d
--- /dev/null
+++ b/libfrog/bitmap.c
@@ -0,0 +1,393 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2018 Oracle.  All Rights Reserved.
+ * Author: Darrick J. Wong <darrick.wong@oracle.com>
+ */
+#include "xfs.h"
+#include <stdint.h>
+#include <stdlib.h>
+#include <assert.h>
+#include <pthread.h>
+#include "platform_defs.h"
+#include "avl64.h"
+#include "list.h"
+#include "bitmap.h"
+
+/*
+ * Space Efficient Bitmap
+ *
+ * Implements a space-efficient bitmap.  We use an AVL tree to manage
+ * extent records that tell us which ranges are set; the bitmap key is
+ * an arbitrary uint64_t.  The usual bitmap operations (set, clear,
+ * test, test and set) are supported, plus we can iterate set ranges.
+ */
+
+#define avl_for_each_range_safe(pos, n, l, first, last) \
+	for (pos = (first), n = pos->avl_nextino, l = (last)->avl_nextino; pos != (l); \
+			pos = n, n = pos ? pos->avl_nextino : NULL)
+
+#define avl_for_each_safe(tree, pos, n) \
+	for (pos = (tree)->avl_firstino, n = pos ? pos->avl_nextino : NULL; \
+			pos != NULL; \
+			pos = n, n = pos ? pos->avl_nextino : NULL)
+
+#define avl_for_each(tree, pos) \
+	for (pos = (tree)->avl_firstino; pos != NULL; pos = pos->avl_nextino)
+
+struct bitmap_node {
+	struct avl64node	btn_node;
+	uint64_t		btn_start;
+	uint64_t		btn_length;
+};
+
+static uint64_t
+extent_start(
+	struct avl64node	*node)
+{
+	struct bitmap_node	*btn;
+
+	btn = container_of(node, struct bitmap_node, btn_node);
+	return btn->btn_start;
+}
+
+static uint64_t
+extent_end(
+	struct avl64node	*node)
+{
+	struct bitmap_node	*btn;
+
+	btn = container_of(node, struct bitmap_node, btn_node);
+	return btn->btn_start + btn->btn_length;
+}
+
+static struct avl64ops bitmap_ops = {
+	extent_start,
+	extent_end,
+};
+
+/* Initialize a bitmap. */
+bool
+bitmap_init(
+	struct bitmap		**bmapp)
+{
+	struct bitmap		*bmap;
+
+	bmap = calloc(1, sizeof(struct bitmap));
+	if (!bmap)
+		return false;
+	bmap->bt_tree = malloc(sizeof(struct avl64tree_desc));
+	if (!bmap->bt_tree) {
+		free(bmap);
+		return false;
+	}
+
+	pthread_mutex_init(&bmap->bt_lock, NULL);
+	avl64_init_tree(bmap->bt_tree, &bitmap_ops);
+	*bmapp = bmap;
+
+	return true;
+}
+
+/* Free a bitmap. */
+void
+bitmap_free(
+	struct bitmap		**bmapp)
+{
+	struct bitmap		*bmap;
+	struct avl64node	*node;
+	struct avl64node	*n;
+	struct bitmap_node	*ext;
+
+	bmap = *bmapp;
+	avl_for_each_safe(bmap->bt_tree, node, n) {
+		ext = container_of(node, struct bitmap_node, btn_node);
+		free(ext);
+	}
+	free(bmap->bt_tree);
+	*bmapp = NULL;
+}
+
+/* Create a new bitmap extent node. */
+static struct bitmap_node *
+bitmap_node_init(
+	uint64_t		start,
+	uint64_t		len)
+{
+	struct bitmap_node	*ext;
+
+	ext = malloc(sizeof(struct bitmap_node));
+	if (!ext)
+		return NULL;
+
+	ext->btn_node.avl_nextino = NULL;
+	ext->btn_start = start;
+	ext->btn_length = len;
+
+	return ext;
+}
+
+/* Set a region of bits (locked). */
+static bool
+__bitmap_set(
+	struct bitmap		*bmap,
+	uint64_t		start,
+	uint64_t		length)
+{
+	struct avl64node	*firstn;
+	struct avl64node	*lastn;
+	struct avl64node	*pos;
+	struct avl64node	*n;
+	struct avl64node	*l;
+	struct bitmap_node	*ext;
+	uint64_t		new_start;
+	uint64_t		new_length;
+	struct avl64node	*node;
+	bool			res = true;
+
+	/* Find any existing nodes adjacent or within that range. */
+	avl64_findranges(bmap->bt_tree, start - 1, start + length + 1,
+			&firstn, &lastn);
+
+	/* Nothing, just insert a new extent. */
+	if (firstn == NULL && lastn == NULL) {
+		ext = bitmap_node_init(start, length);
+		if (!ext)
+			return false;
+
+		node = avl64_insert(bmap->bt_tree, &ext->btn_node);
+		if (node == NULL) {
+			free(ext);
+			errno = EEXIST;
+			return false;
+		}
+
+		return true;
+	}
+
+	assert(firstn != NULL && lastn != NULL);
+	new_start = start;
+	new_length = length;
+
+	avl_for_each_range_safe(pos, n, l, firstn, lastn) {
+		ext = container_of(pos, struct bitmap_node, btn_node);
+
+		/* Bail if the new extent is contained within an old one. */
+		if (ext->btn_start <= start &&
+		    ext->btn_start + ext->btn_length >= start + length)
+			return res;
+
+		/* Check for overlapping and adjacent extents. */
+		if (ext->btn_start + ext->btn_length >= start ||
+		    ext->btn_start <= start + length) {
+			if (ext->btn_start < start) {
+				new_start = ext->btn_start;
+				new_length += ext->btn_length;
+			}
+
+			if (ext->btn_start + ext->btn_length >
+			    new_start + new_length)
+				new_length = ext->btn_start + ext->btn_length -
+						new_start;
+
+			avl64_delete(bmap->bt_tree, pos);
+			free(ext);
+		}
+	}
+
+	ext = bitmap_node_init(new_start, new_length);
+	if (!ext)
+		return false;
+
+	node = avl64_insert(bmap->bt_tree, &ext->btn_node);
+	if (node == NULL) {
+		free(ext);
+		errno = EEXIST;
+		return false;
+	}
+
+	return res;
+}
+
+/* Set a region of bits. */
+bool
+bitmap_set(
+	struct bitmap		*bmap,
+	uint64_t		start,
+	uint64_t		length)
+{
+	bool			res;
+
+	pthread_mutex_lock(&bmap->bt_lock);
+	res = __bitmap_set(bmap, start, length);
+	pthread_mutex_unlock(&bmap->bt_lock);
+
+	return res;
+}
+
+#if 0	/* Unused, provided for completeness. */
+/* Clear a region of bits. */
+bool
+bitmap_clear(
+	struct bitmap		*bmap,
+	uint64_t		start,
+	uint64_t		len)
+{
+	struct avl64node	*firstn;
+	struct avl64node	*lastn;
+	struct avl64node	*pos;
+	struct avl64node	*n;
+	struct avl64node	*l;
+	struct bitmap_node	*ext;
+	uint64_t		new_start;
+	uint64_t		new_length;
+	struct avl64node	*node;
+	int			stat;
+
+	pthread_mutex_lock(&bmap->bt_lock);
+	/* Find any existing nodes over that range. */
+	avl64_findranges(bmap->bt_tree, start, start + len, &firstn, &lastn);
+
+	/* Nothing, we're done. */
+	if (firstn == NULL && lastn == NULL) {
+		pthread_mutex_unlock(&bmap->bt_lock);
+		return true;
+	}
+
+	assert(firstn != NULL && lastn != NULL);
+
+	/* Delete or truncate everything in sight. */
+	avl_for_each_range_safe(pos, n, l, firstn, lastn) {
+		ext = container_of(pos, struct bitmap_node, btn_node);
+
+		stat = 0;
+		if (ext->btn_start < start)
+			stat |= 1;
+		if (ext->btn_start + ext->btn_length > start + len)
+			stat |= 2;
+		switch (stat) {
+		case 0:
+			/* Extent totally within range; delete. */
+			avl64_delete(bmap->bt_tree, pos);
+			free(ext);
+			break;
+		case 1:
+			/* Extent is left-adjacent; truncate. */
+			ext->btn_length = start - ext->btn_start;
+			break;
+		case 2:
+			/* Extent is right-adjacent; move it. */
+			ext->btn_length = ext->btn_start + ext->btn_length -
+					(start + len);
+			ext->btn_start = start + len;
+			break;
+		case 3:
+			/* Extent overlaps both ends. */
+			ext->btn_length = start - ext->btn_start;
+			new_start = start + len;
+			new_length = ext->btn_start + ext->btn_length -
+					new_start;
+
+			ext = bitmap_node_init(new_start, new_length);
+			if (!ext)
+				return false;
+
+			node = avl64_insert(bmap->bt_tree, &ext->btn_node);
+			if (node == NULL) {
+				errno = EEXIST;
+				return false;
+			}
+			break;
+		}
+	}
+
+	pthread_mutex_unlock(&bmap->bt_lock);
+	return true;
+}
+#endif
+
+#ifdef DEBUG
+/* Iterate the set regions of this bitmap. */
+bool
+bitmap_iterate(
+	struct bitmap		*bmap,
+	bool			(*fn)(uint64_t, uint64_t, void *),
+	void			*arg)
+{
+	struct avl64node	*node;
+	struct bitmap_node	*ext;
+	bool			moveon = true;
+
+	pthread_mutex_lock(&bmap->bt_lock);
+	avl_for_each(bmap->bt_tree, node) {
+		ext = container_of(node, struct bitmap_node, btn_node);
+		moveon = fn(ext->btn_start, ext->btn_length, arg);
+		if (!moveon)
+			break;
+	}
+	pthread_mutex_unlock(&bmap->bt_lock);
+
+	return moveon;
+}
+#endif
+
+/* Do any bitmap extents overlap the given one?  (locked) */
+static bool
+__bitmap_test(
+	struct bitmap		*bmap,
+	uint64_t		start,
+	uint64_t		len)
+{
+	struct avl64node	*firstn;
+	struct avl64node	*lastn;
+
+	/* Find any existing nodes over that range. */
+	avl64_findranges(bmap->bt_tree, start, start + len, &firstn, &lastn);
+
+	return firstn != NULL && lastn != NULL;
+}
+
+/* Is any part of this range set? */
+bool
+bitmap_test(
+	struct bitmap		*bmap,
+	uint64_t		start,
+	uint64_t		len)
+{
+	bool			res;
+
+	pthread_mutex_lock(&bmap->bt_lock);
+	res = __bitmap_test(bmap, start, len);
+	pthread_mutex_unlock(&bmap->bt_lock);
+
+	return res;
+}
+
+/* Are none of the bits set? */
+bool
+bitmap_empty(
+	struct bitmap		*bmap)
+{
+	return bmap->bt_tree->avl_firstino == NULL;
+}
+
+#ifdef DEBUG
+static bool
+bitmap_dump_fn(
+	uint64_t		startblock,
+	uint64_t		blockcount,
+	void			*arg)
+{
+	printf("%"PRIu64":%"PRIu64"\n", startblock, blockcount);
+	return true;
+}
+
+/* Dump bitmap. */
+void
+bitmap_dump(
+	struct bitmap		*bmap)
+{
+	printf("BITMAP DUMP %p\n", bmap);
+	bitmap_iterate(bmap, bitmap_dump_fn, NULL);
+	printf("BITMAP DUMP DONE\n");
+}
+#endif
diff --git a/scrub/Makefile b/scrub/Makefile
index bbcfe338..882da8fd 100644
--- a/scrub/Makefile
+++ b/scrub/Makefile
@@ -32,7 +32,6 @@ endif
 endif	# scrub_prereqs
 
 HFILES = \
-bitmap.h \
 common.h \
 counter.h \
 disk.h \
@@ -49,7 +48,6 @@ vfs.h \
 xfs_scrub.h
 
 CFILES = \
-bitmap.c \
 common.c \
 counter.c \
 disk.c \
diff --git a/scrub/bitmap.c b/scrub/bitmap.c
deleted file mode 100644
index aecdba0d..00000000
--- a/scrub/bitmap.c
+++ /dev/null
@@ -1,393 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0+
-/*
- * Copyright (C) 2018 Oracle.  All Rights Reserved.
- * Author: Darrick J. Wong <darrick.wong@oracle.com>
- */
-#include "xfs.h"
-#include <stdint.h>
-#include <stdlib.h>
-#include <assert.h>
-#include <pthread.h>
-#include "platform_defs.h"
-#include "avl64.h"
-#include "list.h"
-#include "bitmap.h"
-
-/*
- * Space Efficient Bitmap
- *
- * Implements a space-efficient bitmap.  We use an AVL tree to manage
- * extent records that tell us which ranges are set; the bitmap key is
- * an arbitrary uint64_t.  The usual bitmap operations (set, clear,
- * test, test and set) are supported, plus we can iterate set ranges.
- */
-
-#define avl_for_each_range_safe(pos, n, l, first, last) \
-	for (pos = (first), n = pos->avl_nextino, l = (last)->avl_nextino; pos != (l); \
-			pos = n, n = pos ? pos->avl_nextino : NULL)
-
-#define avl_for_each_safe(tree, pos, n) \
-	for (pos = (tree)->avl_firstino, n = pos ? pos->avl_nextino : NULL; \
-			pos != NULL; \
-			pos = n, n = pos ? pos->avl_nextino : NULL)
-
-#define avl_for_each(tree, pos) \
-	for (pos = (tree)->avl_firstino; pos != NULL; pos = pos->avl_nextino)
-
-struct bitmap_node {
-	struct avl64node	btn_node;
-	uint64_t		btn_start;
-	uint64_t		btn_length;
-};
-
-static uint64_t
-extent_start(
-	struct avl64node	*node)
-{
-	struct bitmap_node	*btn;
-
-	btn = container_of(node, struct bitmap_node, btn_node);
-	return btn->btn_start;
-}
-
-static uint64_t
-extent_end(
-	struct avl64node	*node)
-{
-	struct bitmap_node	*btn;
-
-	btn = container_of(node, struct bitmap_node, btn_node);
-	return btn->btn_start + btn->btn_length;
-}
-
-static struct avl64ops bitmap_ops = {
-	extent_start,
-	extent_end,
-};
-
-/* Initialize a bitmap. */
-bool
-bitmap_init(
-	struct bitmap		**bmapp)
-{
-	struct bitmap		*bmap;
-
-	bmap = calloc(1, sizeof(struct bitmap));
-	if (!bmap)
-		return false;
-	bmap->bt_tree = malloc(sizeof(struct avl64tree_desc));
-	if (!bmap->bt_tree) {
-		free(bmap);
-		return false;
-	}
-
-	pthread_mutex_init(&bmap->bt_lock, NULL);
-	avl64_init_tree(bmap->bt_tree, &bitmap_ops);
-	*bmapp = bmap;
-
-	return true;
-}
-
-/* Free a bitmap. */
-void
-bitmap_free(
-	struct bitmap		**bmapp)
-{
-	struct bitmap		*bmap;
-	struct avl64node	*node;
-	struct avl64node	*n;
-	struct bitmap_node	*ext;
-
-	bmap = *bmapp;
-	avl_for_each_safe(bmap->bt_tree, node, n) {
-		ext = container_of(node, struct bitmap_node, btn_node);
-		free(ext);
-	}
-	free(bmap->bt_tree);
-	*bmapp = NULL;
-}
-
-/* Create a new bitmap extent node. */
-static struct bitmap_node *
-bitmap_node_init(
-	uint64_t		start,
-	uint64_t		len)
-{
-	struct bitmap_node	*ext;
-
-	ext = malloc(sizeof(struct bitmap_node));
-	if (!ext)
-		return NULL;
-
-	ext->btn_node.avl_nextino = NULL;
-	ext->btn_start = start;
-	ext->btn_length = len;
-
-	return ext;
-}
-
-/* Set a region of bits (locked). */
-static bool
-__bitmap_set(
-	struct bitmap		*bmap,
-	uint64_t		start,
-	uint64_t		length)
-{
-	struct avl64node	*firstn;
-	struct avl64node	*lastn;
-	struct avl64node	*pos;
-	struct avl64node	*n;
-	struct avl64node	*l;
-	struct bitmap_node	*ext;
-	uint64_t		new_start;
-	uint64_t		new_length;
-	struct avl64node	*node;
-	bool			res = true;
-
-	/* Find any existing nodes adjacent or within that range. */
-	avl64_findranges(bmap->bt_tree, start - 1, start + length + 1,
-			&firstn, &lastn);
-
-	/* Nothing, just insert a new extent. */
-	if (firstn == NULL && lastn == NULL) {
-		ext = bitmap_node_init(start, length);
-		if (!ext)
-			return false;
-
-		node = avl64_insert(bmap->bt_tree, &ext->btn_node);
-		if (node == NULL) {
-			free(ext);
-			errno = EEXIST;
-			return false;
-		}
-
-		return true;
-	}
-
-	assert(firstn != NULL && lastn != NULL);
-	new_start = start;
-	new_length = length;
-
-	avl_for_each_range_safe(pos, n, l, firstn, lastn) {
-		ext = container_of(pos, struct bitmap_node, btn_node);
-
-		/* Bail if the new extent is contained within an old one. */
-		if (ext->btn_start <= start &&
-		    ext->btn_start + ext->btn_length >= start + length)
-			return res;
-
-		/* Check for overlapping and adjacent extents. */
-		if (ext->btn_start + ext->btn_length >= start ||
-		    ext->btn_start <= start + length) {
-			if (ext->btn_start < start) {
-				new_start = ext->btn_start;
-				new_length += ext->btn_length;
-			}
-
-			if (ext->btn_start + ext->btn_length >
-			    new_start + new_length)
-				new_length = ext->btn_start + ext->btn_length -
-						new_start;
-
-			avl64_delete(bmap->bt_tree, pos);
-			free(ext);
-		}
-	}
-
-	ext = bitmap_node_init(new_start, new_length);
-	if (!ext)
-		return false;
-
-	node = avl64_insert(bmap->bt_tree, &ext->btn_node);
-	if (node == NULL) {
-		free(ext);
-		errno = EEXIST;
-		return false;
-	}
-
-	return res;
-}
-
-/* Set a region of bits. */
-bool
-bitmap_set(
-	struct bitmap		*bmap,
-	uint64_t		start,
-	uint64_t		length)
-{
-	bool			res;
-
-	pthread_mutex_lock(&bmap->bt_lock);
-	res = __bitmap_set(bmap, start, length);
-	pthread_mutex_unlock(&bmap->bt_lock);
-
-	return res;
-}
-
-#if 0	/* Unused, provided for completeness. */
-/* Clear a region of bits. */
-bool
-bitmap_clear(
-	struct bitmap		*bmap,
-	uint64_t		start,
-	uint64_t		len)
-{
-	struct avl64node	*firstn;
-	struct avl64node	*lastn;
-	struct avl64node	*pos;
-	struct avl64node	*n;
-	struct avl64node	*l;
-	struct bitmap_node	*ext;
-	uint64_t		new_start;
-	uint64_t		new_length;
-	struct avl64node	*node;
-	int			stat;
-
-	pthread_mutex_lock(&bmap->bt_lock);
-	/* Find any existing nodes over that range. */
-	avl64_findranges(bmap->bt_tree, start, start + len, &firstn, &lastn);
-
-	/* Nothing, we're done. */
-	if (firstn == NULL && lastn == NULL) {
-		pthread_mutex_unlock(&bmap->bt_lock);
-		return true;
-	}
-
-	assert(firstn != NULL && lastn != NULL);
-
-	/* Delete or truncate everything in sight. */
-	avl_for_each_range_safe(pos, n, l, firstn, lastn) {
-		ext = container_of(pos, struct bitmap_node, btn_node);
-
-		stat = 0;
-		if (ext->btn_start < start)
-			stat |= 1;
-		if (ext->btn_start + ext->btn_length > start + len)
-			stat |= 2;
-		switch (stat) {
-		case 0:
-			/* Extent totally within range; delete. */
-			avl64_delete(bmap->bt_tree, pos);
-			free(ext);
-			break;
-		case 1:
-			/* Extent is left-adjacent; truncate. */
-			ext->btn_length = start - ext->btn_start;
-			break;
-		case 2:
-			/* Extent is right-adjacent; move it. */
-			ext->btn_length = ext->btn_start + ext->btn_length -
-					(start + len);
-			ext->btn_start = start + len;
-			break;
-		case 3:
-			/* Extent overlaps both ends. */
-			ext->btn_length = start - ext->btn_start;
-			new_start = start + len;
-			new_length = ext->btn_start + ext->btn_length -
-					new_start;
-
-			ext = bitmap_node_init(new_start, new_length);
-			if (!ext)
-				return false;
-
-			node = avl64_insert(bmap->bt_tree, &ext->btn_node);
-			if (node == NULL) {
-				errno = EEXIST;
-				return false;
-			}
-			break;
-		}
-	}
-
-	pthread_mutex_unlock(&bmap->bt_lock);
-	return true;
-}
-#endif
-
-#ifdef DEBUG
-/* Iterate the set regions of this bitmap. */
-bool
-bitmap_iterate(
-	struct bitmap		*bmap,
-	bool			(*fn)(uint64_t, uint64_t, void *),
-	void			*arg)
-{
-	struct avl64node	*node;
-	struct bitmap_node	*ext;
-	bool			moveon = true;
-
-	pthread_mutex_lock(&bmap->bt_lock);
-	avl_for_each(bmap->bt_tree, node) {
-		ext = container_of(node, struct bitmap_node, btn_node);
-		moveon = fn(ext->btn_start, ext->btn_length, arg);
-		if (!moveon)
-			break;
-	}
-	pthread_mutex_unlock(&bmap->bt_lock);
-
-	return moveon;
-}
-#endif
-
-/* Do any bitmap extents overlap the given one?  (locked) */
-static bool
-__bitmap_test(
-	struct bitmap		*bmap,
-	uint64_t		start,
-	uint64_t		len)
-{
-	struct avl64node	*firstn;
-	struct avl64node	*lastn;
-
-	/* Find any existing nodes over that range. */
-	avl64_findranges(bmap->bt_tree, start, start + len, &firstn, &lastn);
-
-	return firstn != NULL && lastn != NULL;
-}
-
-/* Is any part of this range set? */
-bool
-bitmap_test(
-	struct bitmap		*bmap,
-	uint64_t		start,
-	uint64_t		len)
-{
-	bool			res;
-
-	pthread_mutex_lock(&bmap->bt_lock);
-	res = __bitmap_test(bmap, start, len);
-	pthread_mutex_unlock(&bmap->bt_lock);
-
-	return res;
-}
-
-/* Are none of the bits set? */
-bool
-bitmap_empty(
-	struct bitmap		*bmap)
-{
-	return bmap->bt_tree->avl_firstino == NULL;
-}
-
-#ifdef DEBUG
-static bool
-bitmap_dump_fn(
-	uint64_t		startblock,
-	uint64_t		blockcount,
-	void			*arg)
-{
-	printf("%"PRIu64":%"PRIu64"\n", startblock, blockcount);
-	return true;
-}
-
-/* Dump bitmap. */
-void
-bitmap_dump(
-	struct bitmap		*bmap)
-{
-	printf("BITMAP DUMP %p\n", bmap);
-	bitmap_iterate(bmap, bitmap_dump_fn, NULL);
-	printf("BITMAP DUMP DONE\n");
-}
-#endif
diff --git a/scrub/bitmap.h b/scrub/bitmap.h
deleted file mode 100644
index e9746a12..00000000
--- a/scrub/bitmap.h
+++ /dev/null
@@ -1,24 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0+
-/*
- * Copyright (C) 2018 Oracle.  All Rights Reserved.
- * Author: Darrick J. Wong <darrick.wong@oracle.com>
- */
-#ifndef XFS_SCRUB_BITMAP_H_
-#define XFS_SCRUB_BITMAP_H_
-
-struct bitmap {
-	pthread_mutex_t		bt_lock;
-	struct avl64tree_desc	*bt_tree;
-};
-
-bool bitmap_init(struct bitmap **bmap);
-void bitmap_free(struct bitmap **bmap);
-bool bitmap_set(struct bitmap *bmap, uint64_t start, uint64_t length);
-bool bitmap_iterate(struct bitmap *bmap,
-		bool (*fn)(uint64_t, uint64_t, void *), void *arg);
-bool bitmap_test(struct bitmap *bmap, uint64_t start,
-		uint64_t len);
-bool bitmap_empty(struct bitmap *bmap);
-void bitmap_dump(struct bitmap *bmap);
-
-#endif /* XFS_SCRUB_BITMAP_H_ */

Re: [PATCH 28/36] libfrog: hoist bitmap out of scrub

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Move the bitmap code to libfrog so that we can use bitmaps in
> xfs_repair.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

pure move, nothing else

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  include/bitmap.h |   24 +++
>  libfrog/Makefile |    1 
>  libfrog/bitmap.c |  393 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  scrub/Makefile   |    2 
>  scrub/bitmap.c   |  393 ------------------------------------------------------
>  scrub/bitmap.h   |   24 ---
>  6 files changed, 418 insertions(+), 419 deletions(-)
>  create mode 100644 include/bitmap.h
>  create mode 100644 libfrog/bitmap.c
>  delete mode 100644 scrub/bitmap.c
>  delete mode 100644 scrub/bitmap.h
> 
> 
> diff --git a/include/bitmap.h b/include/bitmap.h
> new file mode 100644
> index 00000000..e29a4335
> --- /dev/null
> +++ b/include/bitmap.h
> @@ -0,0 +1,24 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Copyright (C) 2018 Oracle.  All Rights Reserved.
> + * Author: Darrick J. Wong <darrick.wong@oracle.com>
> + */
> +#ifndef LIBFROG_BITMAP_H_
> +#define LIBFROG_BITMAP_H_
> +
> +struct bitmap {
> +	pthread_mutex_t		bt_lock;
> +	struct avl64tree_desc	*bt_tree;
> +};
> +
> +bool bitmap_init(struct bitmap **bmap);
> +void bitmap_free(struct bitmap **bmap);
> +bool bitmap_set(struct bitmap *bmap, uint64_t start, uint64_t length);
> +bool bitmap_iterate(struct bitmap *bmap,
> +		bool (*fn)(uint64_t, uint64_t, void *), void *arg);
> +bool bitmap_test(struct bitmap *bmap, uint64_t start,
> +		uint64_t len);
> +bool bitmap_empty(struct bitmap *bmap);
> +void bitmap_dump(struct bitmap *bmap);
> +
> +#endif /* LIBFROG_BITMAP_H_ */
> diff --git a/libfrog/Makefile b/libfrog/Makefile
> index dbff9596..f5a0539b 100644
> --- a/libfrog/Makefile
> +++ b/libfrog/Makefile
> @@ -12,6 +12,7 @@ LT_AGE = 0
>  
>  CFILES = \
>  avl64.c \
> +bitmap.c \
>  convert.c \
>  crc32.c \
>  fsgeom.c \
> diff --git a/libfrog/bitmap.c b/libfrog/bitmap.c
> new file mode 100644
> index 00000000..aecdba0d
> --- /dev/null
> +++ b/libfrog/bitmap.c
> @@ -0,0 +1,393 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Copyright (C) 2018 Oracle.  All Rights Reserved.
> + * Author: Darrick J. Wong <darrick.wong@oracle.com>
> + */
> +#include "xfs.h"
> +#include <stdint.h>
> +#include <stdlib.h>
> +#include <assert.h>
> +#include <pthread.h>
> +#include "platform_defs.h"
> +#include "avl64.h"
> +#include "list.h"
> +#include "bitmap.h"
> +
> +/*
> + * Space Efficient Bitmap
> + *
> + * Implements a space-efficient bitmap.  We use an AVL tree to manage
> + * extent records that tell us which ranges are set; the bitmap key is
> + * an arbitrary uint64_t.  The usual bitmap operations (set, clear,
> + * test, test and set) are supported, plus we can iterate set ranges.
> + */
> +
> +#define avl_for_each_range_safe(pos, n, l, first, last) \
> +	for (pos = (first), n = pos->avl_nextino, l = (last)->avl_nextino; pos != (l); \
> +			pos = n, n = pos ? pos->avl_nextino : NULL)
> +
> +#define avl_for_each_safe(tree, pos, n) \
> +	for (pos = (tree)->avl_firstino, n = pos ? pos->avl_nextino : NULL; \
> +			pos != NULL; \
> +			pos = n, n = pos ? pos->avl_nextino : NULL)
> +
> +#define avl_for_each(tree, pos) \
> +	for (pos = (tree)->avl_firstino; pos != NULL; pos = pos->avl_nextino)
> +
> +struct bitmap_node {
> +	struct avl64node	btn_node;
> +	uint64_t		btn_start;
> +	uint64_t		btn_length;
> +};
> +
> +static uint64_t
> +extent_start(
> +	struct avl64node	*node)
> +{
> +	struct bitmap_node	*btn;
> +
> +	btn = container_of(node, struct bitmap_node, btn_node);
> +	return btn->btn_start;
> +}
> +
> +static uint64_t
> +extent_end(
> +	struct avl64node	*node)
> +{
> +	struct bitmap_node	*btn;
> +
> +	btn = container_of(node, struct bitmap_node, btn_node);
> +	return btn->btn_start + btn->btn_length;
> +}
> +
> +static struct avl64ops bitmap_ops = {
> +	extent_start,
> +	extent_end,
> +};
> +
> +/* Initialize a bitmap. */
> +bool
> +bitmap_init(
> +	struct bitmap		**bmapp)
> +{
> +	struct bitmap		*bmap;
> +
> +	bmap = calloc(1, sizeof(struct bitmap));
> +	if (!bmap)
> +		return false;
> +	bmap->bt_tree = malloc(sizeof(struct avl64tree_desc));
> +	if (!bmap->bt_tree) {
> +		free(bmap);
> +		return false;
> +	}
> +
> +	pthread_mutex_init(&bmap->bt_lock, NULL);
> +	avl64_init_tree(bmap->bt_tree, &bitmap_ops);
> +	*bmapp = bmap;
> +
> +	return true;
> +}
> +
> +/* Free a bitmap. */
> +void
> +bitmap_free(
> +	struct bitmap		**bmapp)
> +{
> +	struct bitmap		*bmap;
> +	struct avl64node	*node;
> +	struct avl64node	*n;
> +	struct bitmap_node	*ext;
> +
> +	bmap = *bmapp;
> +	avl_for_each_safe(bmap->bt_tree, node, n) {
> +		ext = container_of(node, struct bitmap_node, btn_node);
> +		free(ext);
> +	}
> +	free(bmap->bt_tree);
> +	*bmapp = NULL;
> +}
> +
> +/* Create a new bitmap extent node. */
> +static struct bitmap_node *
> +bitmap_node_init(
> +	uint64_t		start,
> +	uint64_t		len)
> +{
> +	struct bitmap_node	*ext;
> +
> +	ext = malloc(sizeof(struct bitmap_node));
> +	if (!ext)
> +		return NULL;
> +
> +	ext->btn_node.avl_nextino = NULL;
> +	ext->btn_start = start;
> +	ext->btn_length = len;
> +
> +	return ext;
> +}
> +
> +/* Set a region of bits (locked). */
> +static bool
> +__bitmap_set(
> +	struct bitmap		*bmap,
> +	uint64_t		start,
> +	uint64_t		length)
> +{
> +	struct avl64node	*firstn;
> +	struct avl64node	*lastn;
> +	struct avl64node	*pos;
> +	struct avl64node	*n;
> +	struct avl64node	*l;
> +	struct bitmap_node	*ext;
> +	uint64_t		new_start;
> +	uint64_t		new_length;
> +	struct avl64node	*node;
> +	bool			res = true;
> +
> +	/* Find any existing nodes adjacent or within that range. */
> +	avl64_findranges(bmap->bt_tree, start - 1, start + length + 1,
> +			&firstn, &lastn);
> +
> +	/* Nothing, just insert a new extent. */
> +	if (firstn == NULL && lastn == NULL) {
> +		ext = bitmap_node_init(start, length);
> +		if (!ext)
> +			return false;
> +
> +		node = avl64_insert(bmap->bt_tree, &ext->btn_node);
> +		if (node == NULL) {
> +			free(ext);
> +			errno = EEXIST;
> +			return false;
> +		}
> +
> +		return true;
> +	}
> +
> +	assert(firstn != NULL && lastn != NULL);
> +	new_start = start;
> +	new_length = length;
> +
> +	avl_for_each_range_safe(pos, n, l, firstn, lastn) {
> +		ext = container_of(pos, struct bitmap_node, btn_node);
> +
> +		/* Bail if the new extent is contained within an old one. */
> +		if (ext->btn_start <= start &&
> +		    ext->btn_start + ext->btn_length >= start + length)
> +			return res;
> +
> +		/* Check for overlapping and adjacent extents. */
> +		if (ext->btn_start + ext->btn_length >= start ||
> +		    ext->btn_start <= start + length) {
> +			if (ext->btn_start < start) {
> +				new_start = ext->btn_start;
> +				new_length += ext->btn_length;
> +			}
> +
> +			if (ext->btn_start + ext->btn_length >
> +			    new_start + new_length)
> +				new_length = ext->btn_start + ext->btn_length -
> +						new_start;
> +
> +			avl64_delete(bmap->bt_tree, pos);
> +			free(ext);
> +		}
> +	}
> +
> +	ext = bitmap_node_init(new_start, new_length);
> +	if (!ext)
> +		return false;
> +
> +	node = avl64_insert(bmap->bt_tree, &ext->btn_node);
> +	if (node == NULL) {
> +		free(ext);
> +		errno = EEXIST;
> +		return false;
> +	}
> +
> +	return res;
> +}
> +
> +/* Set a region of bits. */
> +bool
> +bitmap_set(
> +	struct bitmap		*bmap,
> +	uint64_t		start,
> +	uint64_t		length)
> +{
> +	bool			res;
> +
> +	pthread_mutex_lock(&bmap->bt_lock);
> +	res = __bitmap_set(bmap, start, length);
> +	pthread_mutex_unlock(&bmap->bt_lock);
> +
> +	return res;
> +}
> +
> +#if 0	/* Unused, provided for completeness. */
> +/* Clear a region of bits. */
> +bool
> +bitmap_clear(
> +	struct bitmap		*bmap,
> +	uint64_t		start,
> +	uint64_t		len)
> +{
> +	struct avl64node	*firstn;
> +	struct avl64node	*lastn;
> +	struct avl64node	*pos;
> +	struct avl64node	*n;
> +	struct avl64node	*l;
> +	struct bitmap_node	*ext;
> +	uint64_t		new_start;
> +	uint64_t		new_length;
> +	struct avl64node	*node;
> +	int			stat;
> +
> +	pthread_mutex_lock(&bmap->bt_lock);
> +	/* Find any existing nodes over that range. */
> +	avl64_findranges(bmap->bt_tree, start, start + len, &firstn, &lastn);
> +
> +	/* Nothing, we're done. */
> +	if (firstn == NULL && lastn == NULL) {
> +		pthread_mutex_unlock(&bmap->bt_lock);
> +		return true;
> +	}
> +
> +	assert(firstn != NULL && lastn != NULL);
> +
> +	/* Delete or truncate everything in sight. */
> +	avl_for_each_range_safe(pos, n, l, firstn, lastn) {
> +		ext = container_of(pos, struct bitmap_node, btn_node);
> +
> +		stat = 0;
> +		if (ext->btn_start < start)
> +			stat |= 1;
> +		if (ext->btn_start + ext->btn_length > start + len)
> +			stat |= 2;
> +		switch (stat) {
> +		case 0:
> +			/* Extent totally within range; delete. */
> +			avl64_delete(bmap->bt_tree, pos);
> +			free(ext);
> +			break;
> +		case 1:
> +			/* Extent is left-adjacent; truncate. */
> +			ext->btn_length = start - ext->btn_start;
> +			break;
> +		case 2:
> +			/* Extent is right-adjacent; move it. */
> +			ext->btn_length = ext->btn_start + ext->btn_length -
> +					(start + len);
> +			ext->btn_start = start + len;
> +			break;
> +		case 3:
> +			/* Extent overlaps both ends. */
> +			ext->btn_length = start - ext->btn_start;
> +			new_start = start + len;
> +			new_length = ext->btn_start + ext->btn_length -
> +					new_start;
> +
> +			ext = bitmap_node_init(new_start, new_length);
> +			if (!ext)
> +				return false;
> +
> +			node = avl64_insert(bmap->bt_tree, &ext->btn_node);
> +			if (node == NULL) {
> +				errno = EEXIST;
> +				return false;
> +			}
> +			break;
> +		}
> +	}
> +
> +	pthread_mutex_unlock(&bmap->bt_lock);
> +	return true;
> +}
> +#endif
> +
> +#ifdef DEBUG
> +/* Iterate the set regions of this bitmap. */
> +bool
> +bitmap_iterate(
> +	struct bitmap		*bmap,
> +	bool			(*fn)(uint64_t, uint64_t, void *),
> +	void			*arg)
> +{
> +	struct avl64node	*node;
> +	struct bitmap_node	*ext;
> +	bool			moveon = true;
> +
> +	pthread_mutex_lock(&bmap->bt_lock);
> +	avl_for_each(bmap->bt_tree, node) {
> +		ext = container_of(node, struct bitmap_node, btn_node);
> +		moveon = fn(ext->btn_start, ext->btn_length, arg);
> +		if (!moveon)
> +			break;
> +	}
> +	pthread_mutex_unlock(&bmap->bt_lock);
> +
> +	return moveon;
> +}
> +#endif
> +
> +/* Do any bitmap extents overlap the given one?  (locked) */
> +static bool
> +__bitmap_test(
> +	struct bitmap		*bmap,
> +	uint64_t		start,
> +	uint64_t		len)
> +{
> +	struct avl64node	*firstn;
> +	struct avl64node	*lastn;
> +
> +	/* Find any existing nodes over that range. */
> +	avl64_findranges(bmap->bt_tree, start, start + len, &firstn, &lastn);
> +
> +	return firstn != NULL && lastn != NULL;
> +}
> +
> +/* Is any part of this range set? */
> +bool
> +bitmap_test(
> +	struct bitmap		*bmap,
> +	uint64_t		start,
> +	uint64_t		len)
> +{
> +	bool			res;
> +
> +	pthread_mutex_lock(&bmap->bt_lock);
> +	res = __bitmap_test(bmap, start, len);
> +	pthread_mutex_unlock(&bmap->bt_lock);
> +
> +	return res;
> +}
> +
> +/* Are none of the bits set? */
> +bool
> +bitmap_empty(
> +	struct bitmap		*bmap)
> +{
> +	return bmap->bt_tree->avl_firstino == NULL;
> +}
> +
> +#ifdef DEBUG
> +static bool
> +bitmap_dump_fn(
> +	uint64_t		startblock,
> +	uint64_t		blockcount,
> +	void			*arg)
> +{
> +	printf("%"PRIu64":%"PRIu64"\n", startblock, blockcount);
> +	return true;
> +}
> +
> +/* Dump bitmap. */
> +void
> +bitmap_dump(
> +	struct bitmap		*bmap)
> +{
> +	printf("BITMAP DUMP %p\n", bmap);
> +	bitmap_iterate(bmap, bitmap_dump_fn, NULL);
> +	printf("BITMAP DUMP DONE\n");
> +}
> +#endif
> diff --git a/scrub/Makefile b/scrub/Makefile
> index bbcfe338..882da8fd 100644
> --- a/scrub/Makefile
> +++ b/scrub/Makefile
> @@ -32,7 +32,6 @@ endif
>  endif	# scrub_prereqs
>  
>  HFILES = \
> -bitmap.h \
>  common.h \
>  counter.h \
>  disk.h \
> @@ -49,7 +48,6 @@ vfs.h \
>  xfs_scrub.h
>  
>  CFILES = \
> -bitmap.c \
>  common.c \
>  counter.c \
>  disk.c \
> diff --git a/scrub/bitmap.c b/scrub/bitmap.c
> deleted file mode 100644
> index aecdba0d..00000000
> --- a/scrub/bitmap.c
> +++ /dev/null
> @@ -1,393 +0,0 @@
> -// SPDX-License-Identifier: GPL-2.0+
> -/*
> - * Copyright (C) 2018 Oracle.  All Rights Reserved.
> - * Author: Darrick J. Wong <darrick.wong@oracle.com>
> - */
> -#include "xfs.h"
> -#include <stdint.h>
> -#include <stdlib.h>
> -#include <assert.h>
> -#include <pthread.h>
> -#include "platform_defs.h"
> -#include "avl64.h"
> -#include "list.h"
> -#include "bitmap.h"
> -
> -/*
> - * Space Efficient Bitmap
> - *
> - * Implements a space-efficient bitmap.  We use an AVL tree to manage
> - * extent records that tell us which ranges are set; the bitmap key is
> - * an arbitrary uint64_t.  The usual bitmap operations (set, clear,
> - * test, test and set) are supported, plus we can iterate set ranges.
> - */
> -
> -#define avl_for_each_range_safe(pos, n, l, first, last) \
> -	for (pos = (first), n = pos->avl_nextino, l = (last)->avl_nextino; pos != (l); \
> -			pos = n, n = pos ? pos->avl_nextino : NULL)
> -
> -#define avl_for_each_safe(tree, pos, n) \
> -	for (pos = (tree)->avl_firstino, n = pos ? pos->avl_nextino : NULL; \
> -			pos != NULL; \
> -			pos = n, n = pos ? pos->avl_nextino : NULL)
> -
> -#define avl_for_each(tree, pos) \
> -	for (pos = (tree)->avl_firstino; pos != NULL; pos = pos->avl_nextino)
> -
> -struct bitmap_node {
> -	struct avl64node	btn_node;
> -	uint64_t		btn_start;
> -	uint64_t		btn_length;
> -};
> -
> -static uint64_t
> -extent_start(
> -	struct avl64node	*node)
> -{
> -	struct bitmap_node	*btn;
> -
> -	btn = container_of(node, struct bitmap_node, btn_node);
> -	return btn->btn_start;
> -}
> -
> -static uint64_t
> -extent_end(
> -	struct avl64node	*node)
> -{
> -	struct bitmap_node	*btn;
> -
> -	btn = container_of(node, struct bitmap_node, btn_node);
> -	return btn->btn_start + btn->btn_length;
> -}
> -
> -static struct avl64ops bitmap_ops = {
> -	extent_start,
> -	extent_end,
> -};
> -
> -/* Initialize a bitmap. */
> -bool
> -bitmap_init(
> -	struct bitmap		**bmapp)
> -{
> -	struct bitmap		*bmap;
> -
> -	bmap = calloc(1, sizeof(struct bitmap));
> -	if (!bmap)
> -		return false;
> -	bmap->bt_tree = malloc(sizeof(struct avl64tree_desc));
> -	if (!bmap->bt_tree) {
> -		free(bmap);
> -		return false;
> -	}
> -
> -	pthread_mutex_init(&bmap->bt_lock, NULL);
> -	avl64_init_tree(bmap->bt_tree, &bitmap_ops);
> -	*bmapp = bmap;
> -
> -	return true;
> -}
> -
> -/* Free a bitmap. */
> -void
> -bitmap_free(
> -	struct bitmap		**bmapp)
> -{
> -	struct bitmap		*bmap;
> -	struct avl64node	*node;
> -	struct avl64node	*n;
> -	struct bitmap_node	*ext;
> -
> -	bmap = *bmapp;
> -	avl_for_each_safe(bmap->bt_tree, node, n) {
> -		ext = container_of(node, struct bitmap_node, btn_node);
> -		free(ext);
> -	}
> -	free(bmap->bt_tree);
> -	*bmapp = NULL;
> -}
> -
> -/* Create a new bitmap extent node. */
> -static struct bitmap_node *
> -bitmap_node_init(
> -	uint64_t		start,
> -	uint64_t		len)
> -{
> -	struct bitmap_node	*ext;
> -
> -	ext = malloc(sizeof(struct bitmap_node));
> -	if (!ext)
> -		return NULL;
> -
> -	ext->btn_node.avl_nextino = NULL;
> -	ext->btn_start = start;
> -	ext->btn_length = len;
> -
> -	return ext;
> -}
> -
> -/* Set a region of bits (locked). */
> -static bool
> -__bitmap_set(
> -	struct bitmap		*bmap,
> -	uint64_t		start,
> -	uint64_t		length)
> -{
> -	struct avl64node	*firstn;
> -	struct avl64node	*lastn;
> -	struct avl64node	*pos;
> -	struct avl64node	*n;
> -	struct avl64node	*l;
> -	struct bitmap_node	*ext;
> -	uint64_t		new_start;
> -	uint64_t		new_length;
> -	struct avl64node	*node;
> -	bool			res = true;
> -
> -	/* Find any existing nodes adjacent or within that range. */
> -	avl64_findranges(bmap->bt_tree, start - 1, start + length + 1,
> -			&firstn, &lastn);
> -
> -	/* Nothing, just insert a new extent. */
> -	if (firstn == NULL && lastn == NULL) {
> -		ext = bitmap_node_init(start, length);
> -		if (!ext)
> -			return false;
> -
> -		node = avl64_insert(bmap->bt_tree, &ext->btn_node);
> -		if (node == NULL) {
> -			free(ext);
> -			errno = EEXIST;
> -			return false;
> -		}
> -
> -		return true;
> -	}
> -
> -	assert(firstn != NULL && lastn != NULL);
> -	new_start = start;
> -	new_length = length;
> -
> -	avl_for_each_range_safe(pos, n, l, firstn, lastn) {
> -		ext = container_of(pos, struct bitmap_node, btn_node);
> -
> -		/* Bail if the new extent is contained within an old one. */
> -		if (ext->btn_start <= start &&
> -		    ext->btn_start + ext->btn_length >= start + length)
> -			return res;
> -
> -		/* Check for overlapping and adjacent extents. */
> -		if (ext->btn_start + ext->btn_length >= start ||
> -		    ext->btn_start <= start + length) {
> -			if (ext->btn_start < start) {
> -				new_start = ext->btn_start;
> -				new_length += ext->btn_length;
> -			}
> -
> -			if (ext->btn_start + ext->btn_length >
> -			    new_start + new_length)
> -				new_length = ext->btn_start + ext->btn_length -
> -						new_start;
> -
> -			avl64_delete(bmap->bt_tree, pos);
> -			free(ext);
> -		}
> -	}
> -
> -	ext = bitmap_node_init(new_start, new_length);
> -	if (!ext)
> -		return false;
> -
> -	node = avl64_insert(bmap->bt_tree, &ext->btn_node);
> -	if (node == NULL) {
> -		free(ext);
> -		errno = EEXIST;
> -		return false;
> -	}
> -
> -	return res;
> -}
> -
> -/* Set a region of bits. */
> -bool
> -bitmap_set(
> -	struct bitmap		*bmap,
> -	uint64_t		start,
> -	uint64_t		length)
> -{
> -	bool			res;
> -
> -	pthread_mutex_lock(&bmap->bt_lock);
> -	res = __bitmap_set(bmap, start, length);
> -	pthread_mutex_unlock(&bmap->bt_lock);
> -
> -	return res;
> -}
> -
> -#if 0	/* Unused, provided for completeness. */
> -/* Clear a region of bits. */
> -bool
> -bitmap_clear(
> -	struct bitmap		*bmap,
> -	uint64_t		start,
> -	uint64_t		len)
> -{
> -	struct avl64node	*firstn;
> -	struct avl64node	*lastn;
> -	struct avl64node	*pos;
> -	struct avl64node	*n;
> -	struct avl64node	*l;
> -	struct bitmap_node	*ext;
> -	uint64_t		new_start;
> -	uint64_t		new_length;
> -	struct avl64node	*node;
> -	int			stat;
> -
> -	pthread_mutex_lock(&bmap->bt_lock);
> -	/* Find any existing nodes over that range. */
> -	avl64_findranges(bmap->bt_tree, start, start + len, &firstn, &lastn);
> -
> -	/* Nothing, we're done. */
> -	if (firstn == NULL && lastn == NULL) {
> -		pthread_mutex_unlock(&bmap->bt_lock);
> -		return true;
> -	}
> -
> -	assert(firstn != NULL && lastn != NULL);
> -
> -	/* Delete or truncate everything in sight. */
> -	avl_for_each_range_safe(pos, n, l, firstn, lastn) {
> -		ext = container_of(pos, struct bitmap_node, btn_node);
> -
> -		stat = 0;
> -		if (ext->btn_start < start)
> -			stat |= 1;
> -		if (ext->btn_start + ext->btn_length > start + len)
> -			stat |= 2;
> -		switch (stat) {
> -		case 0:
> -			/* Extent totally within range; delete. */
> -			avl64_delete(bmap->bt_tree, pos);
> -			free(ext);
> -			break;
> -		case 1:
> -			/* Extent is left-adjacent; truncate. */
> -			ext->btn_length = start - ext->btn_start;
> -			break;
> -		case 2:
> -			/* Extent is right-adjacent; move it. */
> -			ext->btn_length = ext->btn_start + ext->btn_length -
> -					(start + len);
> -			ext->btn_start = start + len;
> -			break;
> -		case 3:
> -			/* Extent overlaps both ends. */
> -			ext->btn_length = start - ext->btn_start;
> -			new_start = start + len;
> -			new_length = ext->btn_start + ext->btn_length -
> -					new_start;
> -
> -			ext = bitmap_node_init(new_start, new_length);
> -			if (!ext)
> -				return false;
> -
> -			node = avl64_insert(bmap->bt_tree, &ext->btn_node);
> -			if (node == NULL) {
> -				errno = EEXIST;
> -				return false;
> -			}
> -			break;
> -		}
> -	}
> -
> -	pthread_mutex_unlock(&bmap->bt_lock);
> -	return true;
> -}
> -#endif
> -
> -#ifdef DEBUG
> -/* Iterate the set regions of this bitmap. */
> -bool
> -bitmap_iterate(
> -	struct bitmap		*bmap,
> -	bool			(*fn)(uint64_t, uint64_t, void *),
> -	void			*arg)
> -{
> -	struct avl64node	*node;
> -	struct bitmap_node	*ext;
> -	bool			moveon = true;
> -
> -	pthread_mutex_lock(&bmap->bt_lock);
> -	avl_for_each(bmap->bt_tree, node) {
> -		ext = container_of(node, struct bitmap_node, btn_node);
> -		moveon = fn(ext->btn_start, ext->btn_length, arg);
> -		if (!moveon)
> -			break;
> -	}
> -	pthread_mutex_unlock(&bmap->bt_lock);
> -
> -	return moveon;
> -}
> -#endif
> -
> -/* Do any bitmap extents overlap the given one?  (locked) */
> -static bool
> -__bitmap_test(
> -	struct bitmap		*bmap,
> -	uint64_t		start,
> -	uint64_t		len)
> -{
> -	struct avl64node	*firstn;
> -	struct avl64node	*lastn;
> -
> -	/* Find any existing nodes over that range. */
> -	avl64_findranges(bmap->bt_tree, start, start + len, &firstn, &lastn);
> -
> -	return firstn != NULL && lastn != NULL;
> -}
> -
> -/* Is any part of this range set? */
> -bool
> -bitmap_test(
> -	struct bitmap		*bmap,
> -	uint64_t		start,
> -	uint64_t		len)
> -{
> -	bool			res;
> -
> -	pthread_mutex_lock(&bmap->bt_lock);
> -	res = __bitmap_test(bmap, start, len);
> -	pthread_mutex_unlock(&bmap->bt_lock);
> -
> -	return res;
> -}
> -
> -/* Are none of the bits set? */
> -bool
> -bitmap_empty(
> -	struct bitmap		*bmap)
> -{
> -	return bmap->bt_tree->avl_firstino == NULL;
> -}
> -
> -#ifdef DEBUG
> -static bool
> -bitmap_dump_fn(
> -	uint64_t		startblock,
> -	uint64_t		blockcount,
> -	void			*arg)
> -{
> -	printf("%"PRIu64":%"PRIu64"\n", startblock, blockcount);
> -	return true;
> -}
> -
> -/* Dump bitmap. */
> -void
> -bitmap_dump(
> -	struct bitmap		*bmap)
> -{
> -	printf("BITMAP DUMP %p\n", bmap);
> -	bitmap_iterate(bmap, bitmap_dump_fn, NULL);
> -	printf("BITMAP DUMP DONE\n");
> -}
> -#endif
> diff --git a/scrub/bitmap.h b/scrub/bitmap.h
> deleted file mode 100644
> index e9746a12..00000000
> --- a/scrub/bitmap.h
> +++ /dev/null
> @@ -1,24 +0,0 @@
> -// SPDX-License-Identifier: GPL-2.0+
> -/*
> - * Copyright (C) 2018 Oracle.  All Rights Reserved.
> - * Author: Darrick J. Wong <darrick.wong@oracle.com>
> - */
> -#ifndef XFS_SCRUB_BITMAP_H_
> -#define XFS_SCRUB_BITMAP_H_
> -
> -struct bitmap {
> -	pthread_mutex_t		bt_lock;
> -	struct avl64tree_desc	*bt_tree;
> -};
> -
> -bool bitmap_init(struct bitmap **bmap);
> -void bitmap_free(struct bitmap **bmap);
> -bool bitmap_set(struct bitmap *bmap, uint64_t start, uint64_t length);
> -bool bitmap_iterate(struct bitmap *bmap,
> -		bool (*fn)(uint64_t, uint64_t, void *), void *arg);
> -bool bitmap_test(struct bitmap *bmap, uint64_t start,
> -		uint64_t len);
> -bool bitmap_empty(struct bitmap *bmap);
> -void bitmap_dump(struct bitmap *bmap);
> -
> -#endif /* XFS_SCRUB_BITMAP_H_ */
> 

[PATCH 29/36] xfs_repair: correctly account for free space btree shrinks when fixing freelist

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

When we fix the freelist at the end of build_agf_agfl in phase 5 of
repair, we need to create incore rmap records for the blocks that get
added to the AGFL.  We can't let the regular freelist fixing code use
the regular on-disk rmapbt update code because the rmapbt isn't fully
set up yet.

Unfortunately, the original code fails to account for the fact that the
free space btrees can shrink when we allocate blocks to fix the
freelist; those blocks are also put on the freelist, but there are
already incore rmaps for all the free space btree blocks.  We must not
create (redundant) incore rmaps for those blocks.  If we do, repair
fails with a complaint that rebuilding the rmapbt failed during phase 5.
xfs/137 on a 1k block size occasionally triggers this bug.

To fix the problem, construct a bitmap of all OWN_AG blocks that we know
about before traversing the AGFL, and only create new incore rmaps for
those AGFL blocks that are not already tracked in the bitmap.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 repair/rmap.c |   54 ++++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 44 insertions(+), 10 deletions(-)


diff --git a/repair/rmap.c b/repair/rmap.c
index d0156f9d..19cceca3 100644
--- a/repair/rmap.c
+++ b/repair/rmap.c
@@ -12,6 +12,7 @@
 #include "dinode.h"
 #include "slab.h"
 #include "rmap.h"
+#include "bitmap.h"
 
 #undef RMAP_DEBUG
 
@@ -450,15 +451,16 @@ rmap_store_ag_btree_rec(
 	struct xfs_buf		*agflbp = NULL;
 	struct xfs_trans	*tp;
 	__be32			*agfl_bno, *b;
+	struct xfs_ag_rmap	*ag_rmap = &ag_rmaps[agno];
+	struct bitmap		*own_ag_bitmap = NULL;
 	int			error = 0;
 
 	if (!xfs_sb_version_hasrmapbt(&mp->m_sb))
 		return 0;
 
 	/* Release the ar_rmaps; they were put into the rmapbt during p5. */
-	free_slab(&ag_rmaps[agno].ar_rmaps);
-	error = init_slab(&ag_rmaps[agno].ar_rmaps,
-				  sizeof(struct xfs_rmap_irec));
+	free_slab(&ag_rmap->ar_rmaps);
+	error = init_slab(&ag_rmap->ar_rmaps, sizeof(struct xfs_rmap_irec));
 	if (error)
 		goto err;
 
@@ -478,19 +480,50 @@ rmap_store_ag_btree_rec(
 	 * rmap, we only need to add rmap records for AGFL blocks past
 	 * that point in the AGFL because those blocks are a result of a
 	 * no-rmap no-shrink freelist fixup that we did earlier.
+	 *
+	 * However, some blocks end up on the AGFL because the free space
+	 * btrees shed blocks as a result of allocating space to fix the
+	 * freelist.  We already created in-core rmap records for the free
+	 * space btree blocks, so we must be careful not to create those
+	 * records again.  Create a bitmap of already-recorded OWN_AG rmaps.
 	 */
+	error = init_slab_cursor(ag_rmap->ar_raw_rmaps, rmap_compare, &rm_cur);
+	if (error)
+		goto err;
+	if (!bitmap_init(&own_ag_bitmap)) {
+		error = -ENOMEM;
+		goto err_slab;
+	}
+	while ((rm_rec = pop_slab_cursor(rm_cur)) != NULL) {
+		if (rm_rec->rm_owner != XFS_RMAP_OWN_AG)
+			continue;
+		if (!bitmap_set(own_ag_bitmap, rm_rec->rm_startblock,
+					rm_rec->rm_blockcount)) {
+			error = EFSCORRUPTED;
+			goto err_slab;
+		}
+	}
+	free_slab_cursor(&rm_cur);
+
+	/* Create rmaps for any AGFL blocks that aren't already rmapped. */
 	agfl_bno = XFS_BUF_TO_AGFL_BNO(mp, agflbp);
-	b = agfl_bno + ag_rmaps[agno].ar_flcount;
+	b = agfl_bno + ag_rmap->ar_flcount;
 	while (*b != cpu_to_be32(NULLAGBLOCK) &&
 	       b - agfl_bno < libxfs_agfl_size(mp)) {
-		error = rmap_add_ag_rec(mp, agno, be32_to_cpu(*b), 1,
-				XFS_RMAP_OWN_AG);
-		if (error)
-			goto err;
+		xfs_agblock_t	agbno;
+
+		agbno = be32_to_cpu(*b);
+		if (!bitmap_test(own_ag_bitmap, agbno, 1)) {
+			error = rmap_add_ag_rec(mp, agno, agbno, 1,
+					XFS_RMAP_OWN_AG);
+			if (error)
+				goto err;
+		}
 		b++;
 	}
 	libxfs_putbuf(agflbp);
 	agflbp = NULL;
+	bitmap_free(&own_ag_bitmap);
 
 	/* Merge all the raw rmaps into the main list */
 	error = rmap_fold_raw_recs(mp, agno);
@@ -498,8 +531,7 @@ rmap_store_ag_btree_rec(
 		goto err;
 
 	/* Create cursors to refcount structures */
-	error = init_slab_cursor(ag_rmaps[agno].ar_rmaps, rmap_compare,
-			&rm_cur);
+	error = init_slab_cursor(ag_rmap->ar_rmaps, rmap_compare, &rm_cur);
 	if (error)
 		goto err;
 
@@ -542,6 +574,8 @@ rmap_store_ag_btree_rec(
 err:
 	if (agflbp)
 		libxfs_putbuf(agflbp);
+	if (own_ag_bitmap)
+		bitmap_free(&own_ag_bitmap);
 	return error;
 }
 

[PATCH 30/36] libxfs: retain ifork_ops when flushing inode

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Retain the ifork ops used to validate the inode so that we can use the
same one to iflush it.  xfs_repair phase 6 can use multiple transactions
to fix various inode problems, which means that the inode might not be
fully fixed when each transaction commits.

This can be a particular problem if there's a shortform directory with
both invalid directory entries and incorrect i8count.  Phase 3 will set
the parent inode to "0" to signal to phase 6 that it needs to reset the
parent and i8count, but phase 6 starts a transaction to junk the bad
entries which fail to commit because the parent is invalid:

fixing i8count in inode 69022994673
Invalid inode number 0x0
xfs_dir_ino_validate: XFS_ERROR_REPORT
Metadata corruption detected at 0x464eb0, inode 0x10121750f1 data fork
xfs_repair: warning - iflush_int failed (-117)

And thus the inode fixes never get written out.

Reported-by: Arkadiusz Miskiewicz <arekm@maven.pl>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 include/xfs_inode.h |    1 +
 libxfs/rdwr.c       |    1 +
 libxfs/util.c       |    2 +-
 3 files changed, 3 insertions(+), 1 deletion(-)


diff --git a/include/xfs_inode.h b/include/xfs_inode.h
index 79ec3a2d..e1e8b430 100644
--- a/include/xfs_inode.h
+++ b/include/xfs_inode.h
@@ -51,6 +51,7 @@ typedef struct xfs_inode {
 
 	xfs_fsize_t		i_size;		/* in-memory size */
 	const struct xfs_dir_ops *d_ops;	/* directory ops vector */
+	struct xfs_ifork_ops	*i_fork_ops;	/* fork verifiers */
 	struct inode		i_vnode;
 } xfs_inode_t;
 
diff --git a/libxfs/rdwr.c b/libxfs/rdwr.c
index a00360e7..69d5abb2 100644
--- a/libxfs/rdwr.c
+++ b/libxfs/rdwr.c
@@ -1391,6 +1391,7 @@ libxfs_iget(
 		return error;
 	}
 
+	ip->i_fork_ops = ifork_ops;
 	if (!libxfs_inode_verify_forks(ip, ifork_ops)) {
 		libxfs_irele(ip);
 		return -EFSCORRUPTED;
diff --git a/libxfs/util.c b/libxfs/util.c
index bd414043..0799f965 100644
--- a/libxfs/util.c
+++ b/libxfs/util.c
@@ -420,7 +420,7 @@ libxfs_iflush_int(xfs_inode_t *ip, xfs_buf_t *bp)
 		VFS_I(ip)->i_version++;
 
 	/* Check the inline fork data before we write out. */
-	if (!libxfs_inode_verify_forks(ip, &xfs_default_ifork_ops))
+	if (!libxfs_inode_verify_forks(ip, ip->i_fork_ops))
 		return -EFSCORRUPTED;
 
 	/*

Re: [PATCH 30/36] libxfs: retain ifork_ops when flushing inode

[Eric Sandeen]

On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Retain the ifork ops used to validate the inode so that we can use the
> same one to iflush it.  xfs_repair phase 6 can use multiple transactions
> to fix various inode problems, which means that the inode might not be
> fully fixed when each transaction commits.
> 
> This can be a particular problem if there's a shortform directory with
> both invalid directory entries and incorrect i8count.  Phase 3 will set
> the parent inode to "0" to signal to phase 6 that it needs to reset the
> parent and i8count, but phase 6 starts a transaction to junk the bad
> entries which fail to commit because the parent is invalid:
> 
> fixing i8count in inode 69022994673
> Invalid inode number 0x0
> xfs_dir_ino_validate: XFS_ERROR_REPORT
> Metadata corruption detected at 0x464eb0, inode 0x10121750f1 data fork
> xfs_repair: warning - iflush_int failed (-117)
> 
> And thus the inode fixes never get written out.

This feels like it would be better to use consistently by dropping the ops
arg to libxfs_inode_verify_forks, using ->i_fork_ops inside it, and setting
it prior to the call.  Er, same for libxfs_iget?

A mishmash of grabbing what was last set in the inode vs. explicitly pointing
at a global ops structure seems a bit ad hoc.  Thoughts?


> 
> Reported-by: Arkadiusz Miskiewicz <arekm@maven.pl>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  include/xfs_inode.h |    1 +
>  libxfs/rdwr.c       |    1 +
>  libxfs/util.c       |    2 +-
>  3 files changed, 3 insertions(+), 1 deletion(-)
> 
> 
> diff --git a/include/xfs_inode.h b/include/xfs_inode.h
> index 79ec3a2d..e1e8b430 100644
> --- a/include/xfs_inode.h
> +++ b/include/xfs_inode.h
> @@ -51,6 +51,7 @@ typedef struct xfs_inode {
>  
>  	xfs_fsize_t		i_size;		/* in-memory size */
>  	const struct xfs_dir_ops *d_ops;	/* directory ops vector */
> +	struct xfs_ifork_ops	*i_fork_ops;	/* fork verifiers */
>  	struct inode		i_vnode;
>  } xfs_inode_t;
>  
> diff --git a/libxfs/rdwr.c b/libxfs/rdwr.c
> index a00360e7..69d5abb2 100644
> --- a/libxfs/rdwr.c
> +++ b/libxfs/rdwr.c
> @@ -1391,6 +1391,7 @@ libxfs_iget(
>  		return error;
>  	}
>  
> +	ip->i_fork_ops = ifork_ops;
>  	if (!libxfs_inode_verify_forks(ip, ifork_ops)) {
>  		libxfs_irele(ip);
>  		return -EFSCORRUPTED;
> diff --git a/libxfs/util.c b/libxfs/util.c
> index bd414043..0799f965 100644
> --- a/libxfs/util.c
> +++ b/libxfs/util.c
> @@ -420,7 +420,7 @@ libxfs_iflush_int(xfs_inode_t *ip, xfs_buf_t *bp)
>  		VFS_I(ip)->i_version++;
>  
>  	/* Check the inline fork data before we write out. */
> -	if (!libxfs_inode_verify_forks(ip, &xfs_default_ifork_ops))
> +	if (!libxfs_inode_verify_forks(ip, ip->i_fork_ops))
>  		return -EFSCORRUPTED;
>  
>  	/*
> 

Re: [PATCH 30/36] libxfs: retain ifork_ops when flushing inode

[Eric Sandeen]

On 4/5/19 1:17 PM, Eric Sandeen wrote:
> On 3/14/19 4:06 PM, Darrick J. Wong wrote:
>> From: Darrick J. Wong <darrick.wong@oracle.com>
>>
>> Retain the ifork ops used to validate the inode so that we can use the
>> same one to iflush it.  xfs_repair phase 6 can use multiple transactions
>> to fix various inode problems, which means that the inode might not be
>> fully fixed when each transaction commits.
>>
>> This can be a particular problem if there's a shortform directory with
>> both invalid directory entries and incorrect i8count.  Phase 3 will set
>> the parent inode to "0" to signal to phase 6 that it needs to reset the
>> parent and i8count, but phase 6 starts a transaction to junk the bad
>> entries which fail to commit because the parent is invalid:
>>
>> fixing i8count in inode 69022994673
>> Invalid inode number 0x0
>> xfs_dir_ino_validate: XFS_ERROR_REPORT
>> Metadata corruption detected at 0x464eb0, inode 0x10121750f1 data fork
>> xfs_repair: warning - iflush_int failed (-117)
>>
>> And thus the inode fixes never get written out.
> 
> This feels like it would be better to use consistently by dropping the ops
> arg to libxfs_inode_verify_forks, using ->i_fork_ops inside it, and setting
> it prior to the call.  Er, same for libxfs_iget?

er, no not for libxfs_iget, sorry.  But libxfs_iget can set the ops on
the inode if found and from then on stuff like libxfs_inode_verify_forks
uses what's currently set in the inode ....)

> A mishmash of grabbing what was last set in the inode vs. explicitly pointing
> at a global ops structure seems a bit ad hoc.  Thoughts?

Re: [PATCH 30/36] libxfs: retain ifork_ops when flushing inode

[Darrick J. Wong]

On Fri, Apr 05, 2019 at 01:19:52PM -0500, Eric Sandeen wrote:
> 
> 
> On 4/5/19 1:17 PM, Eric Sandeen wrote:
> > On 3/14/19 4:06 PM, Darrick J. Wong wrote:
> >> From: Darrick J. Wong <darrick.wong@oracle.com>
> >>
> >> Retain the ifork ops used to validate the inode so that we can use the
> >> same one to iflush it.  xfs_repair phase 6 can use multiple transactions
> >> to fix various inode problems, which means that the inode might not be
> >> fully fixed when each transaction commits.
> >>
> >> This can be a particular problem if there's a shortform directory with
> >> both invalid directory entries and incorrect i8count.  Phase 3 will set
> >> the parent inode to "0" to signal to phase 6 that it needs to reset the
> >> parent and i8count, but phase 6 starts a transaction to junk the bad
> >> entries which fail to commit because the parent is invalid:
> >>
> >> fixing i8count in inode 69022994673
> >> Invalid inode number 0x0
> >> xfs_dir_ino_validate: XFS_ERROR_REPORT
> >> Metadata corruption detected at 0x464eb0, inode 0x10121750f1 data fork
> >> xfs_repair: warning - iflush_int failed (-117)
> >>
> >> And thus the inode fixes never get written out.
> > 
> > This feels like it would be better to use consistently by dropping the ops
> > arg to libxfs_inode_verify_forks, using ->i_fork_ops inside it, and setting
> > it prior to the call.  Er, same for libxfs_iget?
> 
> er, no not for libxfs_iget, sorry.  But libxfs_iget can set the ops on
> the inode if found and from then on stuff like libxfs_inode_verify_forks
> uses what's currently set in the inode ....)
> 
> > A mishmash of grabbing what was last set in the inode vs. explicitly pointing
> > at a global ops structure seems a bit ad hoc.  Thoughts?

Yeah, there's no reason to have the second parameter when we've already
got it in @ip, so I'll make a new patch to refactor it out.

--D

[PATCH 31/36] libxfs: refactor the open-coded libxfs_trans_bjoin calls

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Refactor open-coded bjoin code to use libxfs_trans_bjoin.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 libxfs/trans.c |   18 +++---------------
 1 file changed, 3 insertions(+), 15 deletions(-)


diff --git a/libxfs/trans.c b/libxfs/trans.c
index 10a35dd4..7bdd1544 100644
--- a/libxfs/trans.c
+++ b/libxfs/trans.c
@@ -663,13 +663,9 @@ libxfs_trans_get_buf_map(
 	fprintf(stderr, "trans_get_buf buffer %p, transaction %p\n", bp, tp);
 #endif
 
-	xfs_buf_item_init(bp, tp->t_mountp);
+	libxfs_trans_bjoin(tp, bp);
 	bip = bp->b_log_item;
 	bip->bli_recur = 0;
-	xfs_trans_add_item(tp, (xfs_log_item_t *)bip);
-
-	/* initialize b_transp so we can find it incore */
-	bp->b_transp = tp;
 	return bp;
 }
 
@@ -701,13 +697,9 @@ libxfs_trans_getsb(
 	fprintf(stderr, "trans_get_sb buffer %p, transaction %p\n", bp, tp);
 #endif
 
-	xfs_buf_item_init(bp, mp);
+	libxfs_trans_bjoin(tp, bp);
 	bip = bp->b_log_item;
 	bip->bli_recur = 0;
-	xfs_trans_add_item(tp, (xfs_log_item_t *)bip);
-
-	/* initialize b_transp so we can find it incore */
-	bp->b_transp = tp;
 	return bp;
 }
 
@@ -758,13 +750,9 @@ libxfs_trans_read_buf_map(
 	fprintf(stderr, "trans_read_buf buffer %p, transaction %p\n", bp, tp);
 #endif
 
-	xfs_buf_item_init(bp, tp->t_mountp);
+	xfs_trans_bjoin(tp, bp);
 	bip = bp->b_log_item;
 	bip->bli_recur = 0;
-	xfs_trans_add_item(tp, (xfs_log_item_t *)bip);
-
-	/* initialise b_transp so we can find it incore */
-	bp->b_transp = tp;
 done:
 	*bpp = bp;
 	return 0;

Re: [PATCH 31/36] libxfs: refactor the open-coded libxfs_trans_bjoin calls

[Eric Sandeen]

On 3/14/19 4:07 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Refactor open-coded bjoin code to use libxfs_trans_bjoin.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  libxfs/trans.c |   18 +++---------------
>  1 file changed, 3 insertions(+), 15 deletions(-)
> 
> 
> diff --git a/libxfs/trans.c b/libxfs/trans.c
> index 10a35dd4..7bdd1544 100644
> --- a/libxfs/trans.c
> +++ b/libxfs/trans.c
> @@ -663,13 +663,9 @@ libxfs_trans_get_buf_map(
>  	fprintf(stderr, "trans_get_buf buffer %p, transaction %p\n", bp, tp);
>  #endif
>  
> -	xfs_buf_item_init(bp, tp->t_mountp);
> +	libxfs_trans_bjoin(tp, bp);
>  	bip = bp->b_log_item;
>  	bip->bli_recur = 0;
> -	xfs_trans_add_item(tp, (xfs_log_item_t *)bip);
> -
> -	/* initialize b_transp so we can find it incore */
> -	bp->b_transp = tp;
>  	return bp;
>  }
>  
> @@ -701,13 +697,9 @@ libxfs_trans_getsb(
>  	fprintf(stderr, "trans_get_sb buffer %p, transaction %p\n", bp, tp);
>  #endif
>  
> -	xfs_buf_item_init(bp, mp);
> +	libxfs_trans_bjoin(tp, bp);
>  	bip = bp->b_log_item;
>  	bip->bli_recur = 0;
> -	xfs_trans_add_item(tp, (xfs_log_item_t *)bip);
> -
> -	/* initialize b_transp so we can find it incore */
> -	bp->b_transp = tp;
>  	return bp;
>  }
>  
> @@ -758,13 +750,9 @@ libxfs_trans_read_buf_map(
>  	fprintf(stderr, "trans_read_buf buffer %p, transaction %p\n", bp, tp);
>  #endif
>  
> -	xfs_buf_item_init(bp, tp->t_mountp);
> +	xfs_trans_bjoin(tp, bp);
>  	bip = bp->b_log_item;
>  	bip->bli_recur = 0z
> -	xfs_trans_add_item(tp, (xfs_log_item_t *)bip);
> -
> -	/* initialise b_transp so we can find it incore */
> -	bp->b_transp = tp;
>  done:
>  	*bpp = bp;
>  	return 0;
> 

[PATCH 32/36] libxfs: refactor buffer item release code

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Refactor the buffer item release code into a helper, which we will use
in subsequent patches to make the buffer log item lifetime match the
kernel equivalents.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 libxfs/trans.c |   14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)


diff --git a/libxfs/trans.c b/libxfs/trans.c
index 7bdd1544..295e687e 100644
--- a/libxfs/trans.c
+++ b/libxfs/trans.c
@@ -540,6 +540,16 @@ libxfs_trans_ordered_buf(
 	return ret;
 }
 
+void
+xfs_buf_item_put(
+	struct xfs_buf_log_item	*bip)
+{
+	struct xfs_buf		*bp = bip->bli_buf;
+
+	bp->b_log_item = NULL;
+	kmem_zone_free(xfs_buf_item_zone, bip);
+}
+
 void
 libxfs_trans_brelse(
 	xfs_trans_t		*tp,
@@ -881,7 +891,6 @@ buf_item_done(
 
 	bp = bip->bli_buf;
 	ASSERT(bp != NULL);
-	bp->b_log_item = NULL;			/* remove log item */
 	bp->b_transp = NULL;			/* remove xact ptr */
 
 	hold = (bip->bli_flags & XFS_BLI_HOLD);
@@ -896,8 +905,7 @@ buf_item_done(
 		bip->bli_flags &= ~XFS_BLI_HOLD;
 	else
 		libxfs_putbuf(bp);
-	/* release the buf item */
-	kmem_zone_free(xfs_buf_item_zone, bip);
+	xfs_buf_item_put(bip);
 }
 
 static void

[PATCH 33/36] libxfs: don't touch buffer log item pointer when flushing inode log item

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

When we're flushing an inode log item, it is not necessary to mess with
the inode cluster buffer's log item because the iflush code paths pass
the inode log item directly.  The unconditional reset causes us to leak
buffer log items.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 libxfs/trans.c |    2 --
 1 file changed, 2 deletions(-)


diff --git a/libxfs/trans.c b/libxfs/trans.c
index 295e687e..bd7bbc69 100644
--- a/libxfs/trans.c
+++ b/libxfs/trans.c
@@ -861,10 +861,8 @@ inode_item_done(
 	 * of whether the flush succeed or not. If we fail the flush, make sure
 	 * we still release the buffer reference we currently hold.
 	 */
-	bp->b_log_item = iip;
 	error = libxfs_iflush_int(ip, bp);
 	ip->i_transp = NULL;	/* disassociate from transaction */
-	bp->b_log_item = NULL;	/* remove log item */
 	bp->b_transp = NULL;	/* remove xact ptr */
 
 	if (error) {

[PATCH 34/36] libxfs: fix buffer log item lifetime weirdness

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

In xfsprogs, the lifetime of xfs_buf log items doesn't match the kernel
because we keep them around after comitting or cancelling transactions.
This is confusing, so change the lifetime to be consistent.  Worse yet,
if an inode cluster buffer gets bjoined to a transaction (e.g. someone
called xfs_trans_read_buf) we'll leak it when flushing an inode core
back to that buffer.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 libxfs/trans.c |    2 ++
 1 file changed, 2 insertions(+)


diff --git a/libxfs/trans.c b/libxfs/trans.c
index bd7bbc69..2ceb9838 100644
--- a/libxfs/trans.c
+++ b/libxfs/trans.c
@@ -580,6 +580,7 @@ libxfs_trans_brelse(
 	xfs_trans_del_item(&bip->bli_item);
 	if (bip->bli_flags & XFS_BLI_HOLD)
 		bip->bli_flags &= ~XFS_BLI_HOLD;
+	xfs_buf_item_put(bip);
 	bp->b_transp = NULL;
 	libxfs_putbuf(bp);
 }
@@ -939,6 +940,7 @@ buf_item_unlock(
 
 	hold = bip->bli_flags & XFS_BLI_HOLD;
 	bip->bli_flags &= ~XFS_BLI_HOLD;
+	xfs_buf_item_put(bip);
 	if (!hold)
 		libxfs_putbuf(bp);
 }

[PATCH 35/36] libxfs: shorten inode item lifetime

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Shorten the inode item lifetime so that we only keep them around while
the inode is joined with a transaction.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 libxfs/rdwr.c  |    4 +---
 libxfs/trans.c |   19 ++++++++++++++++---
 2 files changed, 17 insertions(+), 6 deletions(-)


diff --git a/libxfs/rdwr.c b/libxfs/rdwr.c
index 69d5abb2..fd8c6591 100644
--- a/libxfs/rdwr.c
+++ b/libxfs/rdwr.c
@@ -1429,9 +1429,7 @@ void
 libxfs_irele(
 	struct xfs_inode	*ip)
 {
-	if (ip->i_itemp)
-		kmem_zone_free(xfs_ili_zone, ip->i_itemp);
-	ip->i_itemp = NULL;
+	ASSERT(ip->i_itemp == NULL);
 	libxfs_idestroy(ip);
 	kmem_zone_free(xfs_inode_zone, ip);
 }
diff --git a/libxfs/trans.c b/libxfs/trans.c
index 2ceb9838..ce199654 100644
--- a/libxfs/trans.c
+++ b/libxfs/trans.c
@@ -820,6 +820,16 @@ _("Transaction block reservation exceeded! %u > %u\n"),
 	tp->t_flags |= (XFS_TRANS_SB_DIRTY | XFS_TRANS_DIRTY);
 }
 
+void
+xfs_inode_item_put(
+	struct xfs_inode_log_item	*iip)
+{
+	struct xfs_inode		*ip = iip->ili_inode;
+
+	ip->i_itemp = NULL;
+	kmem_zone_free(xfs_ili_zone, iip);
+}
+
 
 /*
  * Transaction commital code follows (i.e. write to disk in libxfs)
@@ -844,7 +854,7 @@ inode_item_done(
 	if (!(iip->ili_fields & XFS_ILOG_ALL)) {
 		ip->i_transp = NULL;	/* disassociate from transaction */
 		iip->ili_flags = 0;	/* reset all flags */
-		return;
+		goto free;
 	}
 
 	/*
@@ -854,7 +864,7 @@ inode_item_done(
 	if (error) {
 		fprintf(stderr, _("%s: warning - imap_to_bp failed (%d)\n"),
 			progname, error);
-		return;
+		goto free;
 	}
 
 	/*
@@ -870,7 +880,7 @@ inode_item_done(
 		fprintf(stderr, _("%s: warning - iflush_int failed (%d)\n"),
 			progname, error);
 		libxfs_putbuf(bp);
-		return;
+		goto free;
 	}
 
 	libxfs_writebuf(bp, 0);
@@ -878,6 +888,8 @@ inode_item_done(
 	fprintf(stderr, "flushing dirty inode %llu, buffer %p\n",
 			ip->i_ino, bp);
 #endif
+free:
+	xfs_inode_item_put(iip);
 }
 
 static void
@@ -955,6 +967,7 @@ inode_item_unlock(
 	ip->i_transp = NULL;
 
 	iip->ili_flags = 0;
+	xfs_inode_item_put(iip);
 }
 
 /* Detach and unlock all of the items in a transaction */

[PATCH 36/36] libxfs: remove libxfs_trans_iget

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

libxfs_trans_iget no longer has a counterpart in the kernel.  Remove it
and make the xfs_iget/xfs_trans_ijoin usage consistent throughout
xfsprogs.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 include/xfs_trans.h      |    2 --
 libxfs/libxfs_api_defs.h |    1 -
 libxfs/trans.c           |   35 -----------------------------------
 libxfs/util.c            |    4 +++-
 repair/phase6.c          |   21 +++++++++++++++------
 repair/phase7.c          |    4 ++--
 6 files changed, 20 insertions(+), 47 deletions(-)


diff --git a/include/xfs_trans.h b/include/xfs_trans.h
index 9e44d18e..e6bb74c4 100644
--- a/include/xfs_trans.h
+++ b/include/xfs_trans.h
@@ -98,8 +98,6 @@ void xfs_defer_cancel(struct xfs_trans *);
 
 struct xfs_buf *libxfs_trans_getsb(struct xfs_trans *, struct xfs_mount *, int);
 
-int	libxfs_trans_iget(struct xfs_mount *, struct xfs_trans *, xfs_ino_t,
-				uint, uint, struct xfs_inode **);
 void	libxfs_trans_ijoin(struct xfs_trans *, struct xfs_inode *, uint);
 void	libxfs_trans_ijoin_ref(struct xfs_trans *, struct xfs_inode *, int);
 void	libxfs_trans_log_inode (struct xfs_trans *, struct xfs_inode *,
diff --git a/libxfs/libxfs_api_defs.h b/libxfs/libxfs_api_defs.h
index e10d78cd..bb0f07bd 100644
--- a/libxfs/libxfs_api_defs.h
+++ b/libxfs/libxfs_api_defs.h
@@ -31,7 +31,6 @@
 #define xfs_trans_del_item		libxfs_trans_del_item
 #define xfs_trans_get_buf		libxfs_trans_get_buf
 #define xfs_trans_getsb			libxfs_trans_getsb
-#define xfs_trans_iget			libxfs_trans_iget
 #define xfs_trans_ichgtime		libxfs_trans_ichgtime
 #define xfs_trans_ijoin			libxfs_trans_ijoin
 #define xfs_trans_init			libxfs_trans_init
diff --git a/libxfs/trans.c b/libxfs/trans.c
index ce199654..d6785e20 100644
--- a/libxfs/trans.c
+++ b/libxfs/trans.c
@@ -336,41 +336,6 @@ libxfs_trans_cancel(
 	return;
 }
 
-int
-libxfs_trans_iget(
-	xfs_mount_t		*mp,
-	xfs_trans_t		*tp,
-	xfs_ino_t		ino,
-	uint			flags,
-	uint			lock_flags,
-	xfs_inode_t		**ipp)
-{
-	int			error;
-	xfs_inode_t		*ip;
-	xfs_inode_log_item_t	*iip;
-
-	if (tp == NULL)
-		return libxfs_iget(mp, tp, ino, lock_flags, ipp,
-				&xfs_default_ifork_ops);
-
-	error = libxfs_iget(mp, tp, ino, lock_flags, &ip,
-			&xfs_default_ifork_ops);
-	if (error)
-		return error;
-	ASSERT(ip != NULL);
-
-	if (ip->i_itemp == NULL)
-		xfs_inode_item_init(ip, mp);
-	iip = ip->i_itemp;
-	xfs_trans_add_item(tp, (xfs_log_item_t *)(iip));
-
-	/* initialize i_transp so we can find it incore */
-	ip->i_transp = tp;
-
-	*ipp = ip;
-	return 0;
-}
-
 void
 libxfs_trans_ijoin(
 	xfs_trans_t		*tp,
diff --git a/libxfs/util.c b/libxfs/util.c
index 0799f965..2e3b9d51 100644
--- a/libxfs/util.c
+++ b/libxfs/util.c
@@ -262,7 +262,8 @@ libxfs_ialloc(
 	}
 	ASSERT(*ialloc_context == NULL);
 
-	error = xfs_trans_iget(tp->t_mountp, tp, ino, 0, 0, &ip);
+	error = libxfs_iget(tp->t_mountp, tp, ino, 0, &ip,
+			&xfs_default_ifork_ops);
 	if (error != 0)
 		return error;
 	ASSERT(ip != NULL);
@@ -376,6 +377,7 @@ libxfs_ialloc(
 	/*
 	 * Log the new values stuffed into the inode.
 	 */
+	xfs_trans_ijoin(tp, ip, 0);
 	xfs_trans_log_inode(tp, ip, flags);
 	*ipp = ip;
 	return 0;
diff --git a/repair/phase6.c b/repair/phase6.c
index 194cfdbf..28e633de 100644
--- a/repair/phase6.c
+++ b/repair/phase6.c
@@ -535,7 +535,8 @@ mk_rbmino(xfs_mount_t *mp)
 	if (i)
 		res_failed(i);
 
-	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rbmino, 0, 0, &ip);
+	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rbmino, 0, &ip,
+			&xfs_default_ifork_ops);
 	if (error) {
 		do_error(
 		_("couldn't iget realtime bitmap inode -- error - %d\n"),
@@ -572,6 +573,7 @@ mk_rbmino(xfs_mount_t *mp)
 	/*
 	 * commit changes
 	 */
+	libxfs_trans_ijoin(tp, ip, 0);
 	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 	error = -libxfs_trans_commit(tp);
 	if (error)
@@ -634,7 +636,8 @@ fill_rbmino(xfs_mount_t *mp)
 	if (error)
 		res_failed(error);
 
-	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rbmino, 0, 0, &ip);
+	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rbmino, 0, &ip,
+			&xfs_default_ifork_ops);
 	if (error) {
 		do_error(
 		_("couldn't iget realtime bitmap inode -- error - %d\n"),
@@ -645,6 +648,7 @@ fill_rbmino(xfs_mount_t *mp)
 		/*
 		 * fill the file one block at a time
 		 */
+		libxfs_trans_ijoin(tp, ip, 0);
 		nmap = 1;
 		error = -libxfs_bmapi_write(tp, ip, bno, 1, 0, 1, &map, &nmap);
 		if (error || nmap != 1) {
@@ -703,7 +707,8 @@ fill_rsumino(xfs_mount_t *mp)
 	if (error)
 		res_failed(error);
 
-	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rsumino, 0, 0, &ip);
+	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rsumino, 0, &ip,
+			&xfs_default_ifork_ops);
 	if (error) {
 		do_error(
 		_("couldn't iget realtime summary inode -- error - %d\n"),
@@ -714,6 +719,7 @@ fill_rsumino(xfs_mount_t *mp)
 		/*
 		 * fill the file one block at a time
 		 */
+		libxfs_trans_ijoin(tp, ip, 0);
 		nmap = 1;
 		error = -libxfs_bmapi_write(tp, ip, bno, 1, 0, 1, &map, &nmap);
 		if (error || nmap != 1) {
@@ -775,7 +781,8 @@ mk_rsumino(xfs_mount_t *mp)
 	if (i)
 		res_failed(i);
 
-	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rsumino, 0, 0, &ip);
+	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rsumino, 0, &ip,
+			&xfs_default_ifork_ops);
 	if (error) {
 		do_error(
 		_("couldn't iget realtime summary inode -- error - %d\n"),
@@ -812,6 +819,7 @@ mk_rsumino(xfs_mount_t *mp)
 	/*
 	 * commit changes
 	 */
+	libxfs_trans_ijoin(tp, ip, 0);
 	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 	error = -libxfs_trans_commit(tp);
 	if (error)
@@ -875,7 +883,8 @@ mk_root_dir(xfs_mount_t *mp)
 	if (i)
 		res_failed(i);
 
-	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rootino, 0, 0, &ip);
+	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rootino, 0, &ip,
+			&xfs_default_ifork_ops);
 	if (error) {
 		do_error(_("could not iget root inode -- error - %d\n"), error);
 	}
@@ -900,7 +909,7 @@ mk_root_dir(xfs_mount_t *mp)
 		times |= XFS_ICHGTIME_CREATE;
 	}
 	libxfs_trans_ichgtime(tp, ip, times);
-
+	libxfs_trans_ijoin(tp, ip, 0);
 	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 
 	/*
diff --git a/repair/phase7.c b/repair/phase7.c
index c2a60a93..c2996470 100644
--- a/repair/phase7.c
+++ b/repair/phase7.c
@@ -32,8 +32,7 @@ update_inode_nlinks(
 	error = -libxfs_trans_alloc(mp, &M_RES(mp)->tr_remove, nres, 0, 0, &tp);
 	ASSERT(error == 0);
 
-	error = -libxfs_trans_iget(mp, tp, ino, 0, 0, &ip);
-
+	error = -libxfs_iget(mp, tp, ino, 0, &ip, &xfs_default_ifork_ops);
 	if (error)  {
 		if (!no_modify)
 			do_error(
@@ -67,6 +66,7 @@ update_inode_nlinks(
 	if (!dirty)  {
 		libxfs_trans_cancel(tp);
 	} else  {
+		libxfs_trans_ijoin(tp, ip, 0);
 		libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 		/*
 		 * no need to do a bmap finish since

Re: [PATCH 36/36] libxfs: remove libxfs_trans_iget

[Eric Sandeen]

On 3/14/19 4:07 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> libxfs_trans_iget no longer has a counterpart in the kernel.  Remove it
> and make the xfs_iget/xfs_trans_ijoin usage consistent throughout
> xfsprogs.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Nice, this syncs us up with roughly 2011 I think ;)

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  include/xfs_trans.h      |    2 --
>  libxfs/libxfs_api_defs.h |    1 -
>  libxfs/trans.c           |   35 -----------------------------------
>  libxfs/util.c            |    4 +++-
>  repair/phase6.c          |   21 +++++++++++++++------
>  repair/phase7.c          |    4 ++--
>  6 files changed, 20 insertions(+), 47 deletions(-)
> 
> 
> diff --git a/include/xfs_trans.h b/include/xfs_trans.h
> index 9e44d18e..e6bb74c4 100644
> --- a/include/xfs_trans.h
> +++ b/include/xfs_trans.h
> @@ -98,8 +98,6 @@ void xfs_defer_cancel(struct xfs_trans *);
>  
>  struct xfs_buf *libxfs_trans_getsb(struct xfs_trans *, struct xfs_mount *, int);
>  
> -int	libxfs_trans_iget(struct xfs_mount *, struct xfs_trans *, xfs_ino_t,
> -				uint, uint, struct xfs_inode **);
>  void	libxfs_trans_ijoin(struct xfs_trans *, struct xfs_inode *, uint);
>  void	libxfs_trans_ijoin_ref(struct xfs_trans *, struct xfs_inode *, int);
>  void	libxfs_trans_log_inode (struct xfs_trans *, struct xfs_inode *,
> diff --git a/libxfs/libxfs_api_defs.h b/libxfs/libxfs_api_defs.h
> index e10d78cd..bb0f07bd 100644
> --- a/libxfs/libxfs_api_defs.h
> +++ b/libxfs/libxfs_api_defs.h
> @@ -31,7 +31,6 @@
>  #define xfs_trans_del_item		libxfs_trans_del_item
>  #define xfs_trans_get_buf		libxfs_trans_get_buf
>  #define xfs_trans_getsb			libxfs_trans_getsb
> -#define xfs_trans_iget			libxfs_trans_iget
>  #define xfs_trans_ichgtime		libxfs_trans_ichgtime
>  #define xfs_trans_ijoin			libxfs_trans_ijoin
>  #define xfs_trans_init			libxfs_trans_init
> diff --git a/libxfs/trans.c b/libxfs/trans.c
> index ce199654..d6785e20 100644
> --- a/libxfs/trans.c
> +++ b/libxfs/trans.c
> @@ -336,41 +336,6 @@ libxfs_trans_cancel(
>  	return;
>  }
>  
> -int
> -libxfs_trans_iget(
> -	xfs_mount_t		*mp,
> -	xfs_trans_t		*tp,
> -	xfs_ino_t		ino,
> -	uint			flags,
> -	uint			lock_flags,
> -	xfs_inode_t		**ipp)
> -{
> -	int			error;
> -	xfs_inode_t		*ip;
> -	xfs_inode_log_item_t	*iip;
> -
> -	if (tp == NULL)
> -		return libxfs_iget(mp, tp, ino, lock_flags, ipp,
> -				&xfs_default_ifork_ops);
> -
> -	error = libxfs_iget(mp, tp, ino, lock_flags, &ip,
> -			&xfs_default_ifork_ops);
> -	if (error)
> -		return error;
> -	ASSERT(ip != NULL);
> -
> -	if (ip->i_itemp == NULL)
> -		xfs_inode_item_init(ip, mp);
> -	iip = ip->i_itemp;
> -	xfs_trans_add_item(tp, (xfs_log_item_t *)(iip));
> -
> -	/* initialize i_transp so we can find it incore */
> -	ip->i_transp = tp;
> -
> -	*ipp = ip;
> -	return 0;
> -}
> -
>  void
>  libxfs_trans_ijoin(
>  	xfs_trans_t		*tp,
> diff --git a/libxfs/util.c b/libxfs/util.c
> index 0799f965..2e3b9d51 100644
> --- a/libxfs/util.c
> +++ b/libxfs/util.c
> @@ -262,7 +262,8 @@ libxfs_ialloc(
>  	}
>  	ASSERT(*ialloc_context == NULL);
>  
> -	error = xfs_trans_iget(tp->t_mountp, tp, ino, 0, 0, &ip);
> +	error = libxfs_iget(tp->t_mountp, tp, ino, 0, &ip,
> +			&xfs_default_ifork_ops);
>  	if (error != 0)
>  		return error;
>  	ASSERT(ip != NULL);
> @@ -376,6 +377,7 @@ libxfs_ialloc(
>  	/*
>  	 * Log the new values stuffed into the inode.
>  	 */
> +	xfs_trans_ijoin(tp, ip, 0);
>  	xfs_trans_log_inode(tp, ip, flags);
>  	*ipp = ip;
>  	return 0;
> diff --git a/repair/phase6.c b/repair/phase6.c
> index 194cfdbf..28e633de 100644
> --- a/repair/phase6.c
> +++ b/repair/phase6.c
> @@ -535,7 +535,8 @@ mk_rbmino(xfs_mount_t *mp)
>  	if (i)
>  		res_failed(i);
>  
> -	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rbmino, 0, 0, &ip);
> +	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rbmino, 0, &ip,
> +			&xfs_default_ifork_ops);
>  	if (error) {
>  		do_error(
>  		_("couldn't iget realtime bitmap inode -- error - %d\n"),
> @@ -572,6 +573,7 @@ mk_rbmino(xfs_mount_t *mp)
>  	/*
>  	 * commit changes
>  	 */
> +	libxfs_trans_ijoin(tp, ip, 0);
>  	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
>  	error = -libxfs_trans_commit(tp);
>  	if (error)
> @@ -634,7 +636,8 @@ fill_rbmino(xfs_mount_t *mp)
>  	if (error)
>  		res_failed(error);
>  
> -	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rbmino, 0, 0, &ip);
> +	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rbmino, 0, &ip,
> +			&xfs_default_ifork_ops);
>  	if (error) {
>  		do_error(
>  		_("couldn't iget realtime bitmap inode -- error - %d\n"),
> @@ -645,6 +648,7 @@ fill_rbmino(xfs_mount_t *mp)
>  		/*
>  		 * fill the file one block at a time
>  		 */
> +		libxfs_trans_ijoin(tp, ip, 0);
>  		nmap = 1;
>  		error = -libxfs_bmapi_write(tp, ip, bno, 1, 0, 1, &map, &nmap);
>  		if (error || nmap != 1) {
> @@ -703,7 +707,8 @@ fill_rsumino(xfs_mount_t *mp)
>  	if (error)
>  		res_failed(error);
>  
> -	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rsumino, 0, 0, &ip);
> +	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rsumino, 0, &ip,
> +			&xfs_default_ifork_ops);
>  	if (error) {
>  		do_error(
>  		_("couldn't iget realtime summary inode -- error - %d\n"),
> @@ -714,6 +719,7 @@ fill_rsumino(xfs_mount_t *mp)
>  		/*
>  		 * fill the file one block at a time
>  		 */
> +		libxfs_trans_ijoin(tp, ip, 0);
>  		nmap = 1;
>  		error = -libxfs_bmapi_write(tp, ip, bno, 1, 0, 1, &map, &nmap);
>  		if (error || nmap != 1) {
> @@ -775,7 +781,8 @@ mk_rsumino(xfs_mount_t *mp)
>  	if (i)
>  		res_failed(i);
>  
> -	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rsumino, 0, 0, &ip);
> +	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rsumino, 0, &ip,
> +			&xfs_default_ifork_ops);
>  	if (error) {
>  		do_error(
>  		_("couldn't iget realtime summary inode -- error - %d\n"),
> @@ -812,6 +819,7 @@ mk_rsumino(xfs_mount_t *mp)
>  	/*
>  	 * commit changes
>  	 */
> +	libxfs_trans_ijoin(tp, ip, 0);
>  	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
>  	error = -libxfs_trans_commit(tp);
>  	if (error)
> @@ -875,7 +883,8 @@ mk_root_dir(xfs_mount_t *mp)
>  	if (i)
>  		res_failed(i);
>  
> -	error = -libxfs_trans_iget(mp, tp, mp->m_sb.sb_rootino, 0, 0, &ip);
> +	error = -libxfs_iget(mp, tp, mp->m_sb.sb_rootino, 0, &ip,
> +			&xfs_default_ifork_ops);
>  	if (error) {
>  		do_error(_("could not iget root inode -- error - %d\n"), error);
>  	}
> @@ -900,7 +909,7 @@ mk_root_dir(xfs_mount_t *mp)
>  		times |= XFS_ICHGTIME_CREATE;
>  	}
>  	libxfs_trans_ichgtime(tp, ip, times);
> -
> +	libxfs_trans_ijoin(tp, ip, 0);
>  	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
>  
>  	/*
> diff --git a/repair/phase7.c b/repair/phase7.c
> index c2a60a93..c2996470 100644
> --- a/repair/phase7.c
> +++ b/repair/phase7.c
> @@ -32,8 +32,7 @@ update_inode_nlinks(
>  	error = -libxfs_trans_alloc(mp, &M_RES(mp)->tr_remove, nres, 0, 0, &tp);
>  	ASSERT(error == 0);
>  
> -	error = -libxfs_trans_iget(mp, tp, ino, 0, 0, &ip);
> -
> +	error = -libxfs_iget(mp, tp, ino, 0, &ip, &xfs_default_ifork_ops);
>  	if (error)  {
>  		if (!no_modify)
>  			do_error(
> @@ -67,6 +66,7 @@ update_inode_nlinks(
>  	if (!dirty)  {
>  		libxfs_trans_cancel(tp);
>  	} else  {
> +		libxfs_trans_ijoin(tp, ip, 0);
>  		libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
>  		/*
>  		 * no need to do a bmap finish since
> 

[PATCH 37/36] xfs_scrub: include unicrash.h in unicrash.c

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix broken function prototype checking in unicrash.c by actually
including its header.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/unicrash.c |    1 +
 1 file changed, 1 insertion(+)

diff --git a/scrub/unicrash.c b/scrub/unicrash.c
index 121eedbc..824b10f0 100644
--- a/scrub/unicrash.c
+++ b/scrub/unicrash.c
@@ -16,6 +16,7 @@
 #include "path.h"
 #include "xfs_scrub.h"
 #include "common.h"
+#include "unicrash.h"
 
 /*
  * Detect Unicode confusable names in directories and attributes.

Re: [PATCH 37/36] xfs_scrub: include unicrash.h in unicrash.c

[Eric Sandeen]

On 3/20/19 2:34 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fix broken function prototype checking in unicrash.c by actually
> including its header.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/unicrash.c |    1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/scrub/unicrash.c b/scrub/unicrash.c
> index 121eedbc..824b10f0 100644
> --- a/scrub/unicrash.c
> +++ b/scrub/unicrash.c
> @@ -16,6 +16,7 @@
>  #include "path.h"
>  #include "xfs_scrub.h"
>  #include "common.h"
> +#include "unicrash.h"
>  
>  /*
>   * Detect Unicode confusable names in directories and attributes.
> 

[PATCH 38/36] xfs_io: don't read garbage stack contents if INUMBERS goes nuts

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

In theory INUMBERS will never return an ocount of zero, but on the off
chance it ever does we'll negative index the igroup array and return
stack contents for an inode number.  Don't do that.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 io/open.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/io/open.c b/io/open.c
index f5fbd2c4..a406ea54 100644
--- a/io/open.c
+++ b/io/open.c
@@ -698,6 +698,9 @@ get_last_inode(void)
 		lastgrp = ocount;
 	}
 
+	if (lastgrp == 0)
+		return 0;
+
 	lastgrp--;
 
 	/* The last inode number in use */

Re: [PATCH 38/36] xfs_io: don't read garbage stack contents if INUMBERS goes nuts

[Eric Sandeen]

On 3/20/19 2:34 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> In theory INUMBERS will never return an ocount of zero, but on the off
> chance it ever does we'll negative index the igroup array and return
> stack contents for an inode number.  Don't do that.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Is there more of a story behind this?  ;)

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  io/open.c |    3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/io/open.c b/io/open.c
> index f5fbd2c4..a406ea54 100644
> --- a/io/open.c
> +++ b/io/open.c
> @@ -698,6 +698,9 @@ get_last_inode(void)
>  		lastgrp = ocount;
>  	}
>  
> +	if (lastgrp == 0)
> +		return 0;
> +
>  	lastgrp--;
>  
>  	/* The last inode number in use */
> 

Re: [PATCH 38/36] xfs_io: don't read garbage stack contents if INUMBERS goes nuts

[Darrick J. Wong]

On Thu, Apr 04, 2019 at 04:12:18PM -0500, Eric Sandeen wrote:
> On 3/20/19 2:34 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > In theory INUMBERS will never return an ocount of zero, but on the off
> > chance it ever does we'll negative index the igroup array and return
> > stack contents for an inode number.  Don't do that.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Is there more of a story behind this?  ;)

/me can't remember, I think it was just shutting up coverity complaints.

--D

> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
> 
> > ---
> >  io/open.c |    3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/io/open.c b/io/open.c
> > index f5fbd2c4..a406ea54 100644
> > --- a/io/open.c
> > +++ b/io/open.c
> > @@ -698,6 +698,9 @@ get_last_inode(void)
> >  		lastgrp = ocount;
> >  	}
> >  
> > +	if (lastgrp == 0)
> > +		return 0;
> > +
> >  	lastgrp--;
> >  
> >  	/* The last inode number in use */
> > 

[PATCH 39/36] misc: fix strncpy length complaints

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix a number of complaints about feeding sizeof(dest) directly to
strncpy.  We do this by declaring the char arrays to be one larger
than necessary and subtracting one, to ensure that we never overfill
the buffer.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 mkfs/xfs_mkfs.c |   13 +++++++++++--
 quota/edit.c    |    9 ++++++---
 2 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
index 9e1c6ec5..e87c692c 100644
--- a/mkfs/xfs_mkfs.c
+++ b/mkfs/xfs_mkfs.c
@@ -3251,8 +3251,17 @@ finish_superblock_setup(
 	struct xfs_mount	*mp,
 	struct xfs_sb		*sbp)
 {
-	if (cfg->label)
-		strncpy(sbp->sb_fname, cfg->label, sizeof(sbp->sb_fname));
+	if (cfg->label) {
+		size_t		label_len;
+
+		/*
+		 * Labels are null terminated unless the string fits exactly
+		 * in the label field, so assume sb_fname is zeroed and then
+		 * do a memcpy because the destination isn't a normal C string.
+		 */
+		label_len = min(sizeof(sbp->sb_fname), strlen(cfg->label));
+		memcpy(sbp->sb_fname, cfg->label, label_len);
+	}
 
 	sbp->sb_dblocks = cfg->dblocks;
 	sbp->sb_rblocks = cfg->rtblocks;
diff --git a/quota/edit.c b/quota/edit.c
index b10a5b34..f9938b8a 100644
--- a/quota/edit.c
+++ b/quota/edit.c
@@ -368,8 +368,7 @@ restore_file(
 	uint		type)
 {
 	char		buffer[512];
-	char		devbuffer[512];
-	char		*dev = NULL;
+	char		dev[512];
 	uint		mask;
 	int		cnt;
 	uint32_t	id;
@@ -377,7 +376,11 @@ restore_file(
 
 	while (fgets(buffer, sizeof(buffer), fp) != NULL) {
 		if (strncmp("fs = ", buffer, 5) == 0) {
-			dev = strncpy(devbuffer, buffer+5, sizeof(devbuffer));
+			/*
+			 * Copy the device name to dev, strip off the trailing
+			 * newline, and move on to the next line.
+			 */
+			strncpy(dev, buffer + 5, sizeof(dev) - 1);
 			dev[strlen(dev) - 1] = '\0';
 			continue;
 		}

Re: [PATCH 39/36] misc: fix strncpy length complaints

[Eric Sandeen]

On 3/20/19 2:35 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fix a number of complaints about feeding sizeof(dest) directly to
> strncpy.  We do this by declaring the char arrays to be one larger
> than necessary and subtracting one, to ensure that we never overfill
> the buffer.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  mkfs/xfs_mkfs.c |   13 +++++++++++--
>  quota/edit.c    |    9 ++++++---
>  2 files changed, 17 insertions(+), 5 deletions(-)
> 
> diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
> index 9e1c6ec5..e87c692c 100644
> --- a/mkfs/xfs_mkfs.c
> +++ b/mkfs/xfs_mkfs.c
> @@ -3251,8 +3251,17 @@ finish_superblock_setup(
>  	struct xfs_mount	*mp,
>  	struct xfs_sb		*sbp)
>  {
> -	if (cfg->label)
> -		strncpy(sbp->sb_fname, cfg->label, sizeof(sbp->sb_fname));
> +	if (cfg->label) {
> +		size_t		label_len;
> +
> +		/*
> +		 * Labels are null terminated unless the string fits exactly
> +		 * in the label field, so assume sb_fname is zeroed and then
> +		 * do a memcpy because the destination isn't a normal C string.
> +		 */
> +		label_len = min(sizeof(sbp->sb_fname), strlen(cfg->label));

Heh, comparing this to the patch I wrote in <checks notes> December and never
sent *cough*, it seems like it might be wise to memset the whole thing to 0,
to be sure we're not leaking bytes here and there in a short label?

+               memset(sbp->sb_fname, 0, sizeof(sbp->sb_fname));

Although, I suppose the prior strncpy had the same issue... *shrug*

> +		memcpy(sbp->sb_fname, cfg->label, label_len);
> +	}
>  
>  	sbp->sb_dblocks = cfg->dblocks;
>  	sbp->sb_rblocks = cfg->rtblocks;
> diff --git a/quota/edit.c b/quota/edit.c
> index b10a5b34..f9938b8a 100644
> --- a/quota/edit.c
> +++ b/quota/edit.c
> @@ -368,8 +368,7 @@ restore_file(
>  	uint		type)
>  {
>  	char		buffer[512];
> -	char		devbuffer[512];
> -	char		*dev = NULL;
> +	char		dev[512];
>  	uint		mask;
>  	int		cnt;
>  	uint32_t	id;
> @@ -377,7 +376,11 @@ restore_file(
>  
>  	while (fgets(buffer, sizeof(buffer), fp) != NULL) {
>  		if (strncmp("fs = ", buffer, 5) == 0) {
> -			dev = strncpy(devbuffer, buffer+5, sizeof(devbuffer));
> +			/*
> +			 * Copy the device name to dev, strip off the trailing
> +			 * newline, and move on to the next line.
> +			 */
> +			strncpy(dev, buffer + 5, sizeof(dev) - 1);
>  			dev[strlen(dev) - 1] = '\0';
>  			continue;
>  		}

Ok.

I had something in io/label.c: label_f() as well.  Do you not get complaints
about strncpy(label, optarg, sizeof(label)); ?

I essentially did the same thing as for mkfs, though with my gratuitous
memset:

                        break;
                case 's':
-                       strncpy(label, optarg, sizeof(label));
+                       len = min(strlen(optarg), sizeof(label));
+                       /* Kernel doesn't care about null termination */
+                       memset(label, 0, sizeof(label));
+                       memcpy(label, optarg, len);
                        break;
                default:

Re: [PATCH 39/36] misc: fix strncpy length complaints

[Darrick J. Wong]

On Thu, Apr 04, 2019 at 04:30:38PM -0500, Eric Sandeen wrote:
> On 3/20/19 2:35 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Fix a number of complaints about feeding sizeof(dest) directly to
> > strncpy.  We do this by declaring the char arrays to be one larger
> > than necessary and subtracting one, to ensure that we never overfill
> > the buffer.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  mkfs/xfs_mkfs.c |   13 +++++++++++--
> >  quota/edit.c    |    9 ++++++---
> >  2 files changed, 17 insertions(+), 5 deletions(-)
> > 
> > diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
> > index 9e1c6ec5..e87c692c 100644
> > --- a/mkfs/xfs_mkfs.c
> > +++ b/mkfs/xfs_mkfs.c
> > @@ -3251,8 +3251,17 @@ finish_superblock_setup(
> >  	struct xfs_mount	*mp,
> >  	struct xfs_sb		*sbp)
> >  {
> > -	if (cfg->label)
> > -		strncpy(sbp->sb_fname, cfg->label, sizeof(sbp->sb_fname));
> > +	if (cfg->label) {
> > +		size_t		label_len;
> > +
> > +		/*
> > +		 * Labels are null terminated unless the string fits exactly
> > +		 * in the label field, so assume sb_fname is zeroed and then
> > +		 * do a memcpy because the destination isn't a normal C string.
> > +		 */
> > +		label_len = min(sizeof(sbp->sb_fname), strlen(cfg->label));
> 
> Heh, comparing this to the patch I wrote in <checks notes> December and never
> sent *cough*, it seems like it might be wise to memset the whole thing to 0,
> to be sure we're not leaking bytes here and there in a short label?
> 
> +               memset(sbp->sb_fname, 0, sizeof(sbp->sb_fname));
> 
> Although, I suppose the prior strncpy had the same issue... *shrug*

I think the initialization is fine because...

finish_superblock_setup sets fields in *sbp.  It gets *sbp from the top
of main(), which does:

	struct xfs_mount	mbuf = {};
	struct xfs_mount	*mp = &mbuf;
	struct xfs_sb		*sbp = &mp->m_sb;

So assuming the empty mbuf initializer causes the whole stack variable
to be initialized to zero, we don't need to do a memset here.

> 
> > +		memcpy(sbp->sb_fname, cfg->label, label_len);
> > +	}
> >  
> >  	sbp->sb_dblocks = cfg->dblocks;
> >  	sbp->sb_rblocks = cfg->rtblocks;
> > diff --git a/quota/edit.c b/quota/edit.c
> > index b10a5b34..f9938b8a 100644
> > --- a/quota/edit.c
> > +++ b/quota/edit.c
> > @@ -368,8 +368,7 @@ restore_file(
> >  	uint		type)
> >  {
> >  	char		buffer[512];
> > -	char		devbuffer[512];
> > -	char		*dev = NULL;
> > +	char		dev[512];
> >  	uint		mask;
> >  	int		cnt;
> >  	uint32_t	id;
> > @@ -377,7 +376,11 @@ restore_file(
> >  
> >  	while (fgets(buffer, sizeof(buffer), fp) != NULL) {
> >  		if (strncmp("fs = ", buffer, 5) == 0) {
> > -			dev = strncpy(devbuffer, buffer+5, sizeof(devbuffer));
> > +			/*
> > +			 * Copy the device name to dev, strip off the trailing
> > +			 * newline, and move on to the next line.
> > +			 */
> > +			strncpy(dev, buffer + 5, sizeof(dev) - 1);
> >  			dev[strlen(dev) - 1] = '\0';
> >  			continue;
> >  		}
> 
> Ok.
> 
> I had something in io/label.c: label_f() as well.  Do you not get complaints
> about strncpy(label, optarg, sizeof(label)); ?

I do, that's why I changed it...?  io/label.c got changed to shut up the
warning about the sizeof(firstarg) as the third parameter.

> 
> I essentially did the same thing as for mkfs, though with my gratuitous
> memset:
> 
>                         break;
>                 case 's':
> -                       strncpy(label, optarg, sizeof(label));
> +                       len = min(strlen(optarg), sizeof(label));
> +                       /* Kernel doesn't care about null termination */
> +                       memset(label, 0, sizeof(label));
> +                       memcpy(label, optarg, len);
>                         break;
>                 default:
> 

(FWIW I'm not sure I remember what we were talking about here.)

Anyway, maybe I'll just resend the remaining patches as a rollup since
this series is already huge.

--D

[PATCH 40/36] xfs_io: fix label parsing and validation

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

When we're trying to set a new label, check the length to make sure we
won't overflow the label size, and size label[] so that we can use
strncpy without static checker complaints.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 io/label.c |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/io/label.c b/io/label.c
index 602ece89..72e07964 100644
--- a/io/label.c
+++ b/io/label.c
@@ -40,7 +40,7 @@ label_f(
 {
 	int		c;
 	int		error;
-	char		label[FSLABEL_MAX];
+	char		label[FSLABEL_MAX + 1];
 
 	if (argc == 1) {
 		memset(label, 0, sizeof(label));
@@ -54,7 +54,13 @@ label_f(
 			label[0] = '\0';
 			break;
 		case 's':
-			strncpy(label, optarg, sizeof(label));
+			if (strlen(optarg) > FSLABEL_MAX) {
+				errno = EINVAL;
+				error = 1;
+				goto out;
+			}
+			strncpy(label, optarg, sizeof(label) - 1);
+			label[sizeof(label) - 1] = 0;
 			break;
 		default:
 			return command_usage(&label_cmd);

Re: [PATCH 40/36] xfs_io: fix label parsing and validation

[Eric Sandeen]

On 3/20/19 2:36 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> When we're trying to set a new label, check the length to make sure we
> won't overflow the label size, and size label[] so that we can use
> strncpy without static checker complaints.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Oh there it is :)  Yeah, this is better than what i had, which
just truncated a too-long label IIRC.

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  io/label.c |   10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/io/label.c b/io/label.c
> index 602ece89..72e07964 100644
> --- a/io/label.c
> +++ b/io/label.c
> @@ -40,7 +40,7 @@ label_f(
>  {
>  	int		c;
>  	int		error;
> -	char		label[FSLABEL_MAX];
> +	char		label[FSLABEL_MAX + 1];
>  
>  	if (argc == 1) {
>  		memset(label, 0, sizeof(label));
> @@ -54,7 +54,13 @@ label_f(
>  			label[0] = '\0';
>  			break;
>  		case 's':
> -			strncpy(label, optarg, sizeof(label));
> +			if (strlen(optarg) > FSLABEL_MAX) {
> +				errno = EINVAL;
> +				error = 1;
> +				goto out;
> +			}
> +			strncpy(label, optarg, sizeof(label) - 1);
> +			label[sizeof(label) - 1] = 0;

Hm, so this can send up to FSLABEL_MAX chars to the kernel w/o a null
termination (the null term is at FSLABEL_MAX+1 right?)

But, I guess that's for the kernel to guard against if it accepts a
label that long, if it even cares.

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

>  			break;
>  		default:
>  			return command_usage(&label_cmd);
> 

[PATCH 41/36] xfs_repair: better cli option parameter checking

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

For the xfs_repair options that require a parameter, check that the user
actually provided one instead of segfaulting.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 repair/xfs_repair.c |   12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/repair/xfs_repair.c b/repair/xfs_repair.c
index 2d333d50..9657503f 100644
--- a/repair/xfs_repair.c
+++ b/repair/xfs_repair.c
@@ -233,10 +233,16 @@ process_args(int argc, char **argv)
 					if (max_mem_specified)
 						do_abort(
 		_("-o bhash option cannot be used with -m option\n"));
+					if (!val)
+						do_abort(
+		_("-o bhash requires a parameter\n"));
 					libxfs_bhash_size = (int)strtol(val, NULL, 0);
 					bhash_option_used = 1;
 					break;
 				case AG_STRIDE:
+					if (!val)
+						do_abort(
+		_("-o ag_stride requires a parameter\n"));
 					ag_stride = (int)strtol(val, NULL, 0);
 					break;
 				case FORCE_GEO:
@@ -247,6 +253,9 @@ process_args(int argc, char **argv)
 					force_geo = 1;
 					break;
 				case PHASE2_THREADS:
+					if (!val)
+						do_abort(
+		_("-o phase2_threads requires a parameter\n"));
 					phase2_threads = (int)strtol(val, NULL, 0);
 					break;
 				default:
@@ -262,6 +271,9 @@ process_args(int argc, char **argv)
 
 				switch (getsubopt(&p, c_opts, &val)) {
 				case CONVERT_LAZY_COUNT:
+					if (!val)
+						do_abort(
+		_("-c lazycount requires a parameter\n"));
 					lazy_count = (int)strtol(val, NULL, 0);
 					convert_lazy_count = 1;
 					break;

Re: [PATCH 41/36] xfs_repair: better cli option parameter checking

[Eric Sandeen]

On 3/20/19 2:36 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> For the xfs_repair options that require a parameter, check that the user
> actually provided one instead of segfaulting.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

looks good, but, uh, neither 

-c lazycount=
nor
-o phase2_threads=

is documented in the man page.  :(

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  repair/xfs_repair.c |   12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/repair/xfs_repair.c b/repair/xfs_repair.c
> index 2d333d50..9657503f 100644
> --- a/repair/xfs_repair.c
> +++ b/repair/xfs_repair.c
> @@ -233,10 +233,16 @@ process_args(int argc, char **argv)
>  					if (max_mem_specified)
>  						do_abort(
>  		_("-o bhash option cannot be used with -m option\n"));
> +					if (!val)
> +						do_abort(
> +		_("-o bhash requires a parameter\n"));
>  					libxfs_bhash_size = (int)strtol(val, NULL, 0);
>  					bhash_option_used = 1;
>  					break;
>  				case AG_STRIDE:
> +					if (!val)
> +						do_abort(
> +		_("-o ag_stride requires a parameter\n"));
>  					ag_stride = (int)strtol(val, NULL, 0);
>  					break;
>  				case FORCE_GEO:
> @@ -247,6 +253,9 @@ process_args(int argc, char **argv)
>  					force_geo = 1;
>  					break;
>  				case PHASE2_THREADS:
> +					if (!val)
> +						do_abort(
> +		_("-o phase2_threads requires a parameter\n"));
>  					phase2_threads = (int)strtol(val, NULL, 0);
>  					break;
>  				default:
> @@ -262,6 +271,9 @@ process_args(int argc, char **argv)
>  
>  				switch (getsubopt(&p, c_opts, &val)) {
>  				case CONVERT_LAZY_COUNT:
> +					if (!val)
> +						do_abort(
> +		_("-c lazycount requires a parameter\n"));
>  					lazy_count = (int)strtol(val, NULL, 0);
>  					convert_lazy_count = 1;
>  					break;
> 

[PATCH 42/36] xfs_db: refactor metadump handling of multi-fsb objects

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Separate the multi-fsb object dispatch from actual dir block processing
so that we can implement symlink handling correctly as a multi-fsb file.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 db/metadump.c |   27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/db/metadump.c b/db/metadump.c
index 3cce3012..83a257d0 100644
--- a/db/metadump.c
+++ b/db/metadump.c
@@ -1911,7 +1911,7 @@ static struct bbmap mfsb_map;
 static int mfsb_length;
 
 static int
-process_multi_fsb_objects(
+process_multi_fsb_dir(
 	xfs_fileoff_t	o,
 	xfs_fsblock_t	s,
 	xfs_filblks_t	c,
@@ -1921,14 +1921,6 @@ process_multi_fsb_objects(
 	char		*dp;
 	int		ret = 0;
 
-	switch (btype) {
-	case TYP_DIR2:
-		break;
-	default:
-		print_warning("bad type for multi-fsb object %d", btype);
-		return -EINVAL;
-	}
-
 	while (c > 0) {
 		unsigned int	bm_len;
 
@@ -1987,6 +1979,23 @@ process_multi_fsb_objects(
 	return ret;
 }
 
+static int
+process_multi_fsb_objects(
+	xfs_fileoff_t	o,
+	xfs_fsblock_t	s,
+	xfs_filblks_t	c,
+	typnm_t		btype,
+	xfs_fileoff_t	last)
+{
+	switch (btype) {
+	case TYP_DIR2:
+		return process_multi_fsb_dir(o, s, c, btype, last);
+	default:
+		print_warning("bad type for multi-fsb object %d", btype);
+		return -EINVAL;
+	}
+}
+
 /* inode copy routines */
 static int
 process_bmbt_reclist(

Re: [PATCH 42/36] xfs_db: refactor metadump handling of multi-fsb objects

[Eric Sandeen]

On 3/20/19 2:36 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Separate the multi-fsb object dispatch from actual dir block processing
> so that we can implement symlink handling correctly as a multi-fsb file.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  db/metadump.c |   27 ++++++++++++++++++---------
>  1 file changed, 18 insertions(+), 9 deletions(-)
> 
> diff --git a/db/metadump.c b/db/metadump.c
> index 3cce3012..83a257d0 100644
> --- a/db/metadump.c
> +++ b/db/metadump.c
> @@ -1911,7 +1911,7 @@ static struct bbmap mfsb_map;
>  static int mfsb_length;
>  
>  static int
> -process_multi_fsb_objects(
> +process_multi_fsb_dir(
>  	xfs_fileoff_t	o,
>  	xfs_fsblock_t	s,
>  	xfs_filblks_t	c,
> @@ -1921,14 +1921,6 @@ process_multi_fsb_objects(
>  	char		*dp;
>  	int		ret = 0;
>  
> -	switch (btype) {
> -	case TYP_DIR2:
> -		break;
> -	default:
> -		print_warning("bad type for multi-fsb object %d", btype);
> -		return -EINVAL;
> -	}
> -
>  	while (c > 0) {
>  		unsigned int	bm_len;
>  
> @@ -1987,6 +1979,23 @@ process_multi_fsb_objects(
>  	return ret;
>  }
>  
> +static int
> +process_multi_fsb_objects(
> +	xfs_fileoff_t	o,
> +	xfs_fsblock_t	s,
> +	xfs_filblks_t	c,
> +	typnm_t		btype,
> +	xfs_fileoff_t	last)
> +{
> +	switch (btype) {
> +	case TYP_DIR2:
> +		return process_multi_fsb_dir(o, s, c, btype, last);
> +	default:
> +		print_warning("bad type for multi-fsb object %d", btype);
> +		return -EINVAL;
> +	}
> +}
> +
>  /* inode copy routines */
>  static int
>  process_bmbt_reclist(
> 

[PATCH 43/36] xfs_db: refactor multi-fsb object detection decision making

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Pull the "is this a multi-fsb object" decision into a separate function
that we can keep close to the actual multi-fsb object dispatcher.  We
will soon make the machinery more complex so we do this to avoid having
a big hairy if statement.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 db/metadump.c |   13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/db/metadump.c b/db/metadump.c
index 83a257d0..57216291 100644
--- a/db/metadump.c
+++ b/db/metadump.c
@@ -1979,6 +1979,16 @@ process_multi_fsb_dir(
 	return ret;
 }
 
+static bool
+is_multi_fsb_object(
+	struct xfs_mount	*mp,
+	typnm_t			btype)
+{
+	if (btype == TYP_DIR2 && mp->m_dir_geo->fsbcount > 1)
+		return true;
+	return false;
+}
+
 static int
 process_multi_fsb_objects(
 	xfs_fileoff_t	o,
@@ -2011,6 +2021,7 @@ process_bmbt_reclist(
 	xfs_fileoff_t		last;
 	xfs_agnumber_t		agno;
 	xfs_agblock_t		agbno;
+	bool			is_multi_fsb = is_multi_fsb_object(mp, btype);
 	int			error;
 
 	if (btype == TYP_DATA)
@@ -2074,7 +2085,7 @@ process_bmbt_reclist(
 		}
 
 		/* multi-extent blocks require special handling */
-		if (btype != TYP_DIR2 || mp->m_dir_geo->fsbcount == 1) {
+		if (!is_multi_fsb) {
 			error = process_single_fsb_objects(o, s, c, btype, last);
 		} else {
 			error = process_multi_fsb_objects(o, s, c, btype, last);

Re: [PATCH 43/36] xfs_db: refactor multi-fsb object detection decision making

[Eric Sandeen]

On 3/20/19 2:37 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Pull the "is this a multi-fsb object" decision into a separate function
> that we can keep close to the actual multi-fsb object dispatcher.  We
> will soon make the machinery more complex so we do this to avoid having
> a big hairy if statement.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  db/metadump.c |   13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/db/metadump.c b/db/metadump.c
> index 83a257d0..57216291 100644
> --- a/db/metadump.c
> +++ b/db/metadump.c
> @@ -1979,6 +1979,16 @@ process_multi_fsb_dir(
>  	return ret;
>  }
>  
> +static bool
> +is_multi_fsb_object(
> +	struct xfs_mount	*mp,
> +	typnm_t			btype)
> +{
> +	if (btype == TYP_DIR2 && mp->m_dir_geo->fsbcount > 1)
> +		return true;
> +	return false;
> +}
> +
>  static int
>  process_multi_fsb_objects(
>  	xfs_fileoff_t	o,
> @@ -2011,6 +2021,7 @@ process_bmbt_reclist(
>  	xfs_fileoff_t		last;
>  	xfs_agnumber_t		agno;
>  	xfs_agblock_t		agbno;
> +	bool			is_multi_fsb = is_multi_fsb_object(mp, btype);
>  	int			error;
>  
>  	if (btype == TYP_DATA)
> @@ -2074,7 +2085,7 @@ process_bmbt_reclist(
>  		}
>  
>  		/* multi-extent blocks require special handling */
> -		if (btype != TYP_DIR2 || mp->m_dir_geo->fsbcount == 1) {
> +		if (!is_multi_fsb) {
>  			error = process_single_fsb_objects(o, s, c, btype, last);
>  		} else {
>  			error = process_multi_fsb_objects(o, s, c, btype, last);
> 

kind of a dumb nitpick, but "if not is" seems kind of backwards, perhaps

		if (is_multi_fsb) {
 			error = process_multi_fsb_objects(o, s, c, btype, last);
 		} else {
 			error = process_single_fsb_objects(o, s, c, btype, last);
 		}

is more natural?

Re: [PATCH 43/36] xfs_db: refactor multi-fsb object detection decision making

[Darrick J. Wong]

On Thu, Apr 04, 2019 at 06:49:10PM -0500, Eric Sandeen wrote:
> On 3/20/19 2:37 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Pull the "is this a multi-fsb object" decision into a separate function
> > that we can keep close to the actual multi-fsb object dispatcher.  We
> > will soon make the machinery more complex so we do this to avoid having
> > a big hairy if statement.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  db/metadump.c |   13 ++++++++++++-
> >  1 file changed, 12 insertions(+), 1 deletion(-)
> > 
> > diff --git a/db/metadump.c b/db/metadump.c
> > index 83a257d0..57216291 100644
> > --- a/db/metadump.c
> > +++ b/db/metadump.c
> > @@ -1979,6 +1979,16 @@ process_multi_fsb_dir(
> >  	return ret;
> >  }
> >  
> > +static bool
> > +is_multi_fsb_object(
> > +	struct xfs_mount	*mp,
> > +	typnm_t			btype)
> > +{
> > +	if (btype == TYP_DIR2 && mp->m_dir_geo->fsbcount > 1)
> > +		return true;
> > +	return false;
> > +}
> > +
> >  static int
> >  process_multi_fsb_objects(
> >  	xfs_fileoff_t	o,
> > @@ -2011,6 +2021,7 @@ process_bmbt_reclist(
> >  	xfs_fileoff_t		last;
> >  	xfs_agnumber_t		agno;
> >  	xfs_agblock_t		agbno;
> > +	bool			is_multi_fsb = is_multi_fsb_object(mp, btype);
> >  	int			error;
> >  
> >  	if (btype == TYP_DATA)
> > @@ -2074,7 +2085,7 @@ process_bmbt_reclist(
> >  		}
> >  
> >  		/* multi-extent blocks require special handling */
> > -		if (btype != TYP_DIR2 || mp->m_dir_geo->fsbcount == 1) {
> > +		if (!is_multi_fsb) {
> >  			error = process_single_fsb_objects(o, s, c, btype, last);
> >  		} else {
> >  			error = process_multi_fsb_objects(o, s, c, btype, last);
> > 
> 
> kind of a dumb nitpick, but "if not is" seems kind of backwards, perhaps
> 
> 		if (is_multi_fsb) {
>  			error = process_multi_fsb_objects(o, s, c, btype, last);
>  		} else {
>  			error = process_single_fsb_objects(o, s, c, btype, last);
>  		}
> 
> is more natural?

Fewest changes necessary, but yes, I will rework that to be less awkward.

--D

[PATCH v2 43/36] xfs_db: refactor multi-fsb object detection decision making

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Pull the "is this a multi-fsb object" decision into a separate function
that we can keep close to the actual multi-fsb object dispatcher.  We
will soon make the machinery more complex so we do this to avoid having
a big hairy if statement.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 db/metadump.c |   22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/db/metadump.c b/db/metadump.c
index 83a257d0..23ecf3b7 100644
--- a/db/metadump.c
+++ b/db/metadump.c
@@ -1979,6 +1979,16 @@ process_multi_fsb_dir(
 	return ret;
 }
 
+static bool
+is_multi_fsb_object(
+	struct xfs_mount	*mp,
+	typnm_t			btype)
+{
+	if (btype == TYP_DIR2 && mp->m_dir_geo->fsbcount > 1)
+		return true;
+	return false;
+}
+
 static int
 process_multi_fsb_objects(
 	xfs_fileoff_t	o,
@@ -2011,6 +2021,7 @@ process_bmbt_reclist(
 	xfs_fileoff_t		last;
 	xfs_agnumber_t		agno;
 	xfs_agblock_t		agbno;
+	bool			is_multi_fsb = is_multi_fsb_object(mp, btype);
 	int			error;
 
 	if (btype == TYP_DATA)
@@ -2074,11 +2085,12 @@ process_bmbt_reclist(
 		}
 
 		/* multi-extent blocks require special handling */
-		if (btype != TYP_DIR2 || mp->m_dir_geo->fsbcount == 1) {
-			error = process_single_fsb_objects(o, s, c, btype, last);
-		} else {
-			error = process_multi_fsb_objects(o, s, c, btype, last);
-		}
+		if (is_multi_fsb)
+			error = process_multi_fsb_objects(o, s, c, btype,
+					last);
+		else
+			error = process_single_fsb_objects(o, s, c, btype,
+					last);
 		if (error)
 			return 0;
 	}

Re: [PATCH v2 43/36] xfs_db: refactor multi-fsb object detection decision making

[Eric Sandeen]

On 4/4/19 7:09 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Pull the "is this a multi-fsb object" decision into a separate function
> that we can keep close to the actual multi-fsb object dispatcher.  We
> will soon make the machinery more complex so we do this to avoid having
> a big hairy if statement.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  db/metadump.c |   22 +++++++++++++++++-----
>  1 file changed, 17 insertions(+), 5 deletions(-)
> 
> diff --git a/db/metadump.c b/db/metadump.c
> index 83a257d0..23ecf3b7 100644
> --- a/db/metadump.c
> +++ b/db/metadump.c
> @@ -1979,6 +1979,16 @@ process_multi_fsb_dir(
>  	return ret;
>  }
>  
> +static bool
> +is_multi_fsb_object(
> +	struct xfs_mount	*mp,
> +	typnm_t			btype)
> +{
> +	if (btype == TYP_DIR2 && mp->m_dir_geo->fsbcount > 1)
> +		return true;
> +	return false;
> +}
> +
>  static int
>  process_multi_fsb_objects(
>  	xfs_fileoff_t	o,
> @@ -2011,6 +2021,7 @@ process_bmbt_reclist(
>  	xfs_fileoff_t		last;
>  	xfs_agnumber_t		agno;
>  	xfs_agblock_t		agbno;
> +	bool			is_multi_fsb = is_multi_fsb_object(mp, btype);
>  	int			error;
>  
>  	if (btype == TYP_DATA)
> @@ -2074,11 +2085,12 @@ process_bmbt_reclist(
>  		}
>  
>  		/* multi-extent blocks require special handling */
> -		if (btype != TYP_DIR2 || mp->m_dir_geo->fsbcount == 1) {
> -			error = process_single_fsb_objects(o, s, c, btype, last);
> -		} else {
> -			error = process_multi_fsb_objects(o, s, c, btype, last);
> -		}
> +		if (is_multi_fsb)
> +			error = process_multi_fsb_objects(o, s, c, btype,
> +					last);
> +		else
> +			error = process_single_fsb_objects(o, s, c, btype,
> +					last);
>  		if (error)
>  			return 0;
>  	}
> 

[PATCH 44/36] xfs_db: metadump should handle symlinks properly

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Remote symlink target blocks are multi-fsb objects on XFS v5 filesystems
because we only write one rmt header per data fork extent.  For fs
blocksize >= 2048 we never have more than one block and therefore nobody
noticed, but for blocksize == 1024 this is definitely not true and leads
to metadump spraying error messages about symlink block crc errors.
Therefore, reformulate the symlink metadump into a multi-fsb dump
function.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 db/metadump.c |   48 +++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)

diff --git a/db/metadump.c b/db/metadump.c
index 57216291..e0dd463f 100644
--- a/db/metadump.c
+++ b/db/metadump.c
@@ -1638,11 +1638,39 @@ process_dir_data_block(
 	}
 }
 
-static void
+static int
 process_symlink_block(
-	char			*block)
+	xfs_fileoff_t	o,
+	xfs_fsblock_t	s,
+	xfs_filblks_t	c,
+	typnm_t		btype,
+	xfs_fileoff_t	last)
 {
-	char *link = block;
+	struct bbmap	map = {
+		.nmaps	= 1,
+		.b	= {
+			{
+				.bm_bn = XFS_FSB_TO_DADDR(mp, s),
+				.bm_len = XFS_FSB_TO_BB(mp, c),
+			}
+		},
+	};
+	char		*link;
+	int		ret = 0;
+
+	push_cur();
+	set_cur(&typtab[btype], 0, 0, DB_RING_IGN, &map);
+	if (!iocur_top->data) {
+		xfs_agnumber_t	agno = XFS_FSB_TO_AGNO(mp, s);
+		xfs_agblock_t	agbno = XFS_FSB_TO_AGBNO(mp, s);
+
+		print_warning("cannot read %s block %u/%u (%llu)",
+				typtab[btype].name, agno, agbno, s);
+		if (stop_on_read_error)
+			ret = -1;
+		goto out_pop;
+	}
+	link = iocur_top->data;
 
 	if (xfs_sb_version_hascrc(&(mp)->m_sb))
 		link += sizeof(struct xfs_dsymlink_hdr);
@@ -1660,6 +1688,12 @@ process_symlink_block(
 		if (zlen < mp->m_sb.sb_blocksize)
 			memset(link + linklen, 0, zlen);
 	}
+
+	iocur_top->need_crc = 1;
+	ret = write_buf(iocur_top);
+out_pop:
+	pop_cur();
+	return ret;
 }
 
 #define MAX_REMOTE_VALS		4095
@@ -1879,10 +1913,6 @@ process_single_fsb_objects(
 			}
 			iocur_top->need_crc = 1;
 			break;
-		case TYP_SYMLINK:
-			process_symlink_block(dp);
-			iocur_top->need_crc = 1;
-			break;
 		case TYP_ATTR:
 			process_attr_block(dp, o);
 			iocur_top->need_crc = 1;
@@ -1986,6 +2016,8 @@ is_multi_fsb_object(
 {
 	if (btype == TYP_DIR2 && mp->m_dir_geo->fsbcount > 1)
 		return true;
+	if (btype == TYP_SYMLINK)
+		return true;
 	return false;
 }
 
@@ -2000,6 +2032,8 @@ process_multi_fsb_objects(
 	switch (btype) {
 	case TYP_DIR2:
 		return process_multi_fsb_dir(o, s, c, btype, last);
+	case TYP_SYMLINK:
+		return process_symlink_block(o, s, c, btype, last);
 	default:
 		print_warning("bad type for multi-fsb object %d", btype);
 		return -EINVAL;

Re: [PATCH 44/36] xfs_db: metadump should handle symlinks properly

[Eric Sandeen]

On 3/20/19 2:37 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Remote symlink target blocks are multi-fsb objects on XFS v5 filesystems
> because we only write one rmt header per data fork extent.  For fs
> blocksize >= 2048 we never have more than one block and therefore nobody
> noticed, but for blocksize == 1024 this is definitely not true and leads
> to metadump spraying error messages about symlink block crc errors.
> Therefore, reformulate the symlink metadump into a multi-fsb dump
> function.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Ok, thanks for the patient re-explanation on IRC, and after a bit of testing,
even with fragmented multi-block symlinks, this does work fine.

I'd kind of like to fully understand why, with a single map, it handles
fragmented links, but for now, moving on ... 

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

I'd like to change the struct bbmap initialization to outside the
declaration, though, to be more in line with other similar code.
Any heartburn over that?

	map.nmaps = 1;
	map.b[0].bm_bn = XFS_FSB_TO_DADDR(mp, s);
	map.b[0].bm_len = XFS_FSB_TO_BB(mp, c);



> ---
>  db/metadump.c |   48 +++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 41 insertions(+), 7 deletions(-)
> 
> diff --git a/db/metadump.c b/db/metadump.c
> index 57216291..e0dd463f 100644
> --- a/db/metadump.c
> +++ b/db/metadump.c
> @@ -1638,11 +1638,39 @@ process_dir_data_block(
>  	}
>  }
>  
> -static void
> +static int
>  process_symlink_block(
> -	char			*block)
> +	xfs_fileoff_t	o,
> +	xfs_fsblock_t	s,
> +	xfs_filblks_t	c,
> +	typnm_t		btype,
> +	xfs_fileoff_t	last)
>  {
> -	char *link = block;
> +	struct bbmap	map = {
> +		.nmaps	= 1,
> +		.b	= {
> +			{
> +				.bm_bn = XFS_FSB_TO_DADDR(mp, s),
> +				.bm_len = XFS_FSB_TO_BB(mp, c),
> +			}
> +		},
> +	};
> +	char		*link;
> +	int		ret = 0;
> +
> +	push_cur();
> +	set_cur(&typtab[btype], 0, 0, DB_RING_IGN, &map);
> +	if (!iocur_top->data) {
> +		xfs_agnumber_t	agno = XFS_FSB_TO_AGNO(mp, s);
> +		xfs_agblock_t	agbno = XFS_FSB_TO_AGBNO(mp, s);
> +
> +		print_warning("cannot read %s block %u/%u (%llu)",
> +				typtab[btype].name, agno, agbno, s);
> +		if (stop_on_read_error)
> +			ret = -1;
> +		goto out_pop;
> +	}
> +	link = iocur_top->data;
>  
>  	if (xfs_sb_version_hascrc(&(mp)->m_sb))
>  		link += sizeof(struct xfs_dsymlink_hdr);
> @@ -1660,6 +1688,12 @@ process_symlink_block(
>  		if (zlen < mp->m_sb.sb_blocksize)
>  			memset(link + linklen, 0, zlen);
>  	}
> +
> +	iocur_top->need_crc = 1;
> +	ret = write_buf(iocur_top);
> +out_pop:
> +	pop_cur();
> +	return ret;
>  }
>  
>  #define MAX_REMOTE_VALS		4095
> @@ -1879,10 +1913,6 @@ process_single_fsb_objects(
>  			}
>  			iocur_top->need_crc = 1;
>  			break;
> -		case TYP_SYMLINK:
> -			process_symlink_block(dp);
> -			iocur_top->need_crc = 1;
> -			break;
>  		case TYP_ATTR:
>  			process_attr_block(dp, o);
>  			iocur_top->need_crc = 1;
> @@ -1986,6 +2016,8 @@ is_multi_fsb_object(
>  {
>  	if (btype == TYP_DIR2 && mp->m_dir_geo->fsbcount > 1)
>  		return true;
> +	if (btype == TYP_SYMLINK)
> +		return true;
>  	return false;
>  }
>  
> @@ -2000,6 +2032,8 @@ process_multi_fsb_objects(
>  	switch (btype) {
>  	case TYP_DIR2:
>  		return process_multi_fsb_dir(o, s, c, btype, last);
> +	case TYP_SYMLINK:
> +		return process_symlink_block(o, s, c, btype, last);
>  	default:
>  		print_warning("bad type for multi-fsb object %d", btype);
>  		return -EINVAL;
> 

Re: [PATCH 44/36] xfs_db: metadump should handle symlinks properly

[Darrick J. Wong]

On Fri, Apr 05, 2019 at 09:18:58AM -0500, Eric Sandeen wrote:
> On 3/20/19 2:37 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Remote symlink target blocks are multi-fsb objects on XFS v5 filesystems
> > because we only write one rmt header per data fork extent.  For fs
> > blocksize >= 2048 we never have more than one block and therefore nobody
> > noticed, but for blocksize == 1024 this is definitely not true and leads
> > to metadump spraying error messages about symlink block crc errors.
> > Therefore, reformulate the symlink metadump into a multi-fsb dump
> > function.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Ok, thanks for the patient re-explanation on IRC, and after a bit of testing,
> even with fragmented multi-block symlinks, this does work fine.
> 
> I'd kind of like to fully understand why, with a single map, it handles
> fragmented links, but for now, moving on ... 
> 
> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
> 
> I'd like to change the struct bbmap initialization to outside the
> declaration, though, to be more in line with other similar code.
> Any heartburn over that?
> 
> 	map.nmaps = 1;
> 	map.b[0].bm_bn = XFS_FSB_TO_DADDR(mp, s);
> 	map.b[0].bm_len = XFS_FSB_TO_BB(mp, c);

No, that's fine.  Admittedly that thing I wrote is five lines longer
than it needs to be; I'll write myself a BMW, etc.  Feel free to change
it. :)

--D

> 
> 
> 
> > ---
> >  db/metadump.c |   48 +++++++++++++++++++++++++++++++++++++++++-------
> >  1 file changed, 41 insertions(+), 7 deletions(-)
> > 
> > diff --git a/db/metadump.c b/db/metadump.c
> > index 57216291..e0dd463f 100644
> > --- a/db/metadump.c
> > +++ b/db/metadump.c
> > @@ -1638,11 +1638,39 @@ process_dir_data_block(
> >  	}
> >  }
> >  
> > -static void
> > +static int
> >  process_symlink_block(
> > -	char			*block)
> > +	xfs_fileoff_t	o,
> > +	xfs_fsblock_t	s,
> > +	xfs_filblks_t	c,
> > +	typnm_t		btype,
> > +	xfs_fileoff_t	last)
> >  {
> > -	char *link = block;
> > +	struct bbmap	map = {
> > +		.nmaps	= 1,
> > +		.b	= {
> > +			{
> > +				.bm_bn = XFS_FSB_TO_DADDR(mp, s),
> > +				.bm_len = XFS_FSB_TO_BB(mp, c),
> > +			}
> > +		},
> > +	};
> > +	char		*link;
> > +	int		ret = 0;
> > +
> > +	push_cur();
> > +	set_cur(&typtab[btype], 0, 0, DB_RING_IGN, &map);
> > +	if (!iocur_top->data) {
> > +		xfs_agnumber_t	agno = XFS_FSB_TO_AGNO(mp, s);
> > +		xfs_agblock_t	agbno = XFS_FSB_TO_AGBNO(mp, s);
> > +
> > +		print_warning("cannot read %s block %u/%u (%llu)",
> > +				typtab[btype].name, agno, agbno, s);
> > +		if (stop_on_read_error)
> > +			ret = -1;
> > +		goto out_pop;
> > +	}
> > +	link = iocur_top->data;
> >  
> >  	if (xfs_sb_version_hascrc(&(mp)->m_sb))
> >  		link += sizeof(struct xfs_dsymlink_hdr);
> > @@ -1660,6 +1688,12 @@ process_symlink_block(
> >  		if (zlen < mp->m_sb.sb_blocksize)
> >  			memset(link + linklen, 0, zlen);
> >  	}
> > +
> > +	iocur_top->need_crc = 1;
> > +	ret = write_buf(iocur_top);
> > +out_pop:
> > +	pop_cur();
> > +	return ret;
> >  }
> >  
> >  #define MAX_REMOTE_VALS		4095
> > @@ -1879,10 +1913,6 @@ process_single_fsb_objects(
> >  			}
> >  			iocur_top->need_crc = 1;
> >  			break;
> > -		case TYP_SYMLINK:
> > -			process_symlink_block(dp);
> > -			iocur_top->need_crc = 1;
> > -			break;
> >  		case TYP_ATTR:
> >  			process_attr_block(dp, o);
> >  			iocur_top->need_crc = 1;
> > @@ -1986,6 +2016,8 @@ is_multi_fsb_object(
> >  {
> >  	if (btype == TYP_DIR2 && mp->m_dir_geo->fsbcount > 1)
> >  		return true;
> > +	if (btype == TYP_SYMLINK)
> > +		return true;
> >  	return false;
> >  }
> >  
> > @@ -2000,6 +2032,8 @@ process_multi_fsb_objects(
> >  	switch (btype) {
> >  	case TYP_DIR2:
> >  		return process_multi_fsb_dir(o, s, c, btype, last);
> > +	case TYP_SYMLINK:
> > +		return process_symlink_block(o, s, c, btype, last);
> >  	default:
> >  		print_warning("bad type for multi-fsb object %d", btype);
> >  		return -EINVAL;
> > 

[PATCH 45/36] xfs_scrub: rename confusing structure

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Rename xfs_verify_extent to media_verify_state so that the name more
accurately reflects what we're storing in the structure.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase6.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/scrub/phase6.c b/scrub/phase6.c
index 5e2b9ee3..2b4f6ff1 100644
--- a/scrub/phase6.c
+++ b/scrub/phase6.c
@@ -35,7 +35,7 @@
 
 /* Verify disk blocks with GETFSMAP */
 
-struct xfs_verify_extent {
+struct media_verify_state {
 	struct read_verify_pool	*rvp_data;
 	struct read_verify_pool	*rvp_log;
 	struct read_verify_pool	*rvp_realtime;
@@ -47,7 +47,7 @@ struct xfs_verify_extent {
 static struct read_verify_pool *
 xfs_dev_to_pool(
 	struct scrub_ctx		*ctx,
-	struct xfs_verify_extent	*ve,
+	struct media_verify_state	*ve,
 	dev_t				dev)
 {
 	if (dev == ctx->fsinfo.fs_datadev)
@@ -348,7 +348,7 @@ xfs_check_rmap_ioerr(
 {
 	struct fsmap			keys[2];
 	char				descr[DESCR_BUFSZ];
-	struct xfs_verify_extent	*ve = arg;
+	struct media_verify_state	*ve = arg;
 	struct bitmap			*tree;
 	dev_t				dev;
 	bool				moveon;
@@ -395,7 +395,7 @@ xfs_check_rmap(
 	struct fsmap			*map,
 	void				*arg)
 {
-	struct xfs_verify_extent	*ve = arg;
+	struct media_verify_state	*ve = arg;
 	struct read_verify_pool		*rvp;
 
 	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
@@ -463,7 +463,7 @@ bool
 xfs_scan_blocks(
 	struct scrub_ctx		*ctx)
 {
-	struct xfs_verify_extent	ve = { NULL };
+	struct media_verify_state	ve = { NULL };
 	bool				moveon;
 
 	moveon = bitmap_init(&ve.d_bad);

Re: [PATCH 45/36] xfs_scrub: rename confusing structure

[Eric Sandeen]

On 3/20/19 3:03 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Rename xfs_verify_extent to media_verify_state so that the name more
> accurately reflects what we're storing in the structure.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Hate to do this, but "*ve" made sense for "verify extent" but
really doesn't for "verify state..."  If the point is clarity, I
think s/ve/vs/ would be reasonable as well?  It's about 35 instances.

> ---
>  scrub/phase6.c |   10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/scrub/phase6.c b/scrub/phase6.c
> index 5e2b9ee3..2b4f6ff1 100644
> --- a/scrub/phase6.c
> +++ b/scrub/phase6.c
> @@ -35,7 +35,7 @@
>  
>  /* Verify disk blocks with GETFSMAP */
>  
> -struct xfs_verify_extent {
> +struct media_verify_state {
>  	struct read_verify_pool	*rvp_data;
>  	struct read_verify_pool	*rvp_log;
>  	struct read_verify_pool	*rvp_realtime;
> @@ -47,7 +47,7 @@ struct xfs_verify_extent {
>  static struct read_verify_pool *
>  xfs_dev_to_pool(
>  	struct scrub_ctx		*ctx,
> -	struct xfs_verify_extent	*ve,
> +	struct media_verify_state	*ve,
>  	dev_t				dev)
>  {
>  	if (dev == ctx->fsinfo.fs_datadev)
> @@ -348,7 +348,7 @@ xfs_check_rmap_ioerr(
>  {
>  	struct fsmap			keys[2];
>  	char				descr[DESCR_BUFSZ];
> -	struct xfs_verify_extent	*ve = arg;
> +	struct media_verify_state	*ve = arg;
>  	struct bitmap			*tree;
>  	dev_t				dev;
>  	bool				moveon;
> @@ -395,7 +395,7 @@ xfs_check_rmap(
>  	struct fsmap			*map,
>  	void				*arg)
>  {
> -	struct xfs_verify_extent	*ve = arg;
> +	struct media_verify_state	*ve = arg;
>  	struct read_verify_pool		*rvp;
>  
>  	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
> @@ -463,7 +463,7 @@ bool
>  xfs_scan_blocks(
>  	struct scrub_ctx		*ctx)
>  {
> -	struct xfs_verify_extent	ve = { NULL };
> +	struct media_verify_state	ve = { NULL };
>  	bool				moveon;
>  
>  	moveon = bitmap_init(&ve.d_bad);
> 

Re: [PATCH 45/36] xfs_scrub: rename confusing structure

[Darrick J. Wong]

On Thu, Apr 04, 2019 at 11:40:53AM -0500, Eric Sandeen wrote:
> On 3/20/19 3:03 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Rename xfs_verify_extent to media_verify_state so that the name more
> > accurately reflects what we're storing in the structure.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Hate to do this, but "*ve" made sense for "verify extent" but
> really doesn't for "verify state..."  If the point is clarity, I
> think s/ve/vs/ would be reasonable as well?  It's about 35 instances.

Sure.  I couldn't come up with a synonym for 'state' that starts with
an 'e'. :/

--D

> > ---
> >  scrub/phase6.c |   10 +++++-----
> >  1 file changed, 5 insertions(+), 5 deletions(-)
> > 
> > diff --git a/scrub/phase6.c b/scrub/phase6.c
> > index 5e2b9ee3..2b4f6ff1 100644
> > --- a/scrub/phase6.c
> > +++ b/scrub/phase6.c
> > @@ -35,7 +35,7 @@
> >  
> >  /* Verify disk blocks with GETFSMAP */
> >  
> > -struct xfs_verify_extent {
> > +struct media_verify_state {
> >  	struct read_verify_pool	*rvp_data;
> >  	struct read_verify_pool	*rvp_log;
> >  	struct read_verify_pool	*rvp_realtime;
> > @@ -47,7 +47,7 @@ struct xfs_verify_extent {
> >  static struct read_verify_pool *
> >  xfs_dev_to_pool(
> >  	struct scrub_ctx		*ctx,
> > -	struct xfs_verify_extent	*ve,
> > +	struct media_verify_state	*ve,
> >  	dev_t				dev)
> >  {
> >  	if (dev == ctx->fsinfo.fs_datadev)
> > @@ -348,7 +348,7 @@ xfs_check_rmap_ioerr(
> >  {
> >  	struct fsmap			keys[2];
> >  	char				descr[DESCR_BUFSZ];
> > -	struct xfs_verify_extent	*ve = arg;
> > +	struct media_verify_state	*ve = arg;
> >  	struct bitmap			*tree;
> >  	dev_t				dev;
> >  	bool				moveon;
> > @@ -395,7 +395,7 @@ xfs_check_rmap(
> >  	struct fsmap			*map,
> >  	void				*arg)
> >  {
> > -	struct xfs_verify_extent	*ve = arg;
> > +	struct media_verify_state	*ve = arg;
> >  	struct read_verify_pool		*rvp;
> >  
> >  	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
> > @@ -463,7 +463,7 @@ bool
> >  xfs_scan_blocks(
> >  	struct scrub_ctx		*ctx)
> >  {
> > -	struct xfs_verify_extent	ve = { NULL };
> > +	struct media_verify_state	ve = { NULL };
> >  	bool				moveon;
> >  
> >  	moveon = bitmap_init(&ve.d_bad);
> > 

[PATCH v2 45/36] xfs_scrub: rename confusing structure

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Rename xfs_verify_extent to media_verify_state so that the name more
accurately reflects what we're storing in the structure.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase6.c |   70 ++++++++++++++++++++++++++++----------------------------
 1 file changed, 35 insertions(+), 35 deletions(-)

diff --git a/scrub/phase6.c b/scrub/phase6.c
index 5e2b9ee3..91e4443e 100644
--- a/scrub/phase6.c
+++ b/scrub/phase6.c
@@ -35,7 +35,7 @@
 
 /* Verify disk blocks with GETFSMAP */
 
-struct xfs_verify_extent {
+struct media_verify_state {
 	struct read_verify_pool	*rvp_data;
 	struct read_verify_pool	*rvp_log;
 	struct read_verify_pool	*rvp_realtime;
@@ -47,15 +47,15 @@ struct xfs_verify_extent {
 static struct read_verify_pool *
 xfs_dev_to_pool(
 	struct scrub_ctx		*ctx,
-	struct xfs_verify_extent	*ve,
+	struct media_verify_state	*vs,
 	dev_t				dev)
 {
 	if (dev == ctx->fsinfo.fs_datadev)
-		return ve->rvp_data;
+		return vs->rvp_data;
 	else if (dev == ctx->fsinfo.fs_logdev)
-		return ve->rvp_log;
+		return vs->rvp_log;
 	else if (dev == ctx->fsinfo.fs_rtdev)
-		return ve->rvp_realtime;
+		return vs->rvp_realtime;
 	abort();
 }
 
@@ -348,7 +348,7 @@ xfs_check_rmap_ioerr(
 {
 	struct fsmap			keys[2];
 	char				descr[DESCR_BUFSZ];
-	struct xfs_verify_extent	*ve = arg;
+	struct media_verify_state	*vs = arg;
 	struct bitmap			*tree;
 	dev_t				dev;
 	bool				moveon;
@@ -360,9 +360,9 @@ xfs_check_rmap_ioerr(
 	 * later rescanning.
 	 */
 	if (dev == ctx->fsinfo.fs_datadev)
-		tree = ve->d_bad;
+		tree = vs->d_bad;
 	else if (dev == ctx->fsinfo.fs_rtdev)
-		tree = ve->r_bad;
+		tree = vs->r_bad;
 	else
 		tree = NULL;
 	if (tree) {
@@ -395,10 +395,10 @@ xfs_check_rmap(
 	struct fsmap			*map,
 	void				*arg)
 {
-	struct xfs_verify_extent	*ve = arg;
+	struct media_verify_state	*vs = arg;
 	struct read_verify_pool		*rvp;
 
-	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
+	rvp = xfs_dev_to_pool(ctx, vs, map->fmr_device);
 
 	dbg_printf("rmap dev %d:%d phys %"PRIu64" owner %"PRId64
 			" offset %"PRIu64" len %"PRIu64" flags 0x%x\n",
@@ -425,7 +425,7 @@ xfs_check_rmap(
 	/* XXX: Filter out directory data blocks. */
 
 	/* Schedule the read verify command for (eventual) running. */
-	read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, ve);
+	read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, vs);
 
 out:
 	/* Is this the last extent?  Fire off the read. */
@@ -463,35 +463,35 @@ bool
 xfs_scan_blocks(
 	struct scrub_ctx		*ctx)
 {
-	struct xfs_verify_extent	ve = { NULL };
+	struct media_verify_state	vs = { NULL };
 	bool				moveon;
 
-	moveon = bitmap_init(&ve.d_bad);
+	moveon = bitmap_init(&vs.d_bad);
 	if (!moveon) {
 		str_errno(ctx, ctx->mntpoint);
 		goto out;
 	}
 
-	moveon = bitmap_init(&ve.r_bad);
+	moveon = bitmap_init(&vs.r_bad);
 	if (!moveon) {
 		str_errno(ctx, ctx->mntpoint);
 		goto out_dbad;
 	}
 
-	ve.rvp_data = read_verify_pool_init(ctx, ctx->datadev,
+	vs.rvp_data = read_verify_pool_init(ctx, ctx->datadev,
 			ctx->geo.blocksize, xfs_check_rmap_ioerr,
 			scrub_nproc(ctx));
-	if (!ve.rvp_data) {
+	if (!vs.rvp_data) {
 		moveon = false;
 		str_info(ctx, ctx->mntpoint,
 _("Could not create data device media verifier."));
 		goto out_rbad;
 	}
 	if (ctx->logdev) {
-		ve.rvp_log = read_verify_pool_init(ctx, ctx->logdev,
+		vs.rvp_log = read_verify_pool_init(ctx, ctx->logdev,
 				ctx->geo.blocksize, xfs_check_rmap_ioerr,
 				scrub_nproc(ctx));
-		if (!ve.rvp_log) {
+		if (!vs.rvp_log) {
 			moveon = false;
 			str_info(ctx, ctx->mntpoint,
 	_("Could not create log device media verifier."));
@@ -499,43 +499,43 @@ _("Could not create data device media verifier."));
 		}
 	}
 	if (ctx->rtdev) {
-		ve.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev,
+		vs.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev,
 				ctx->geo.blocksize, xfs_check_rmap_ioerr,
 				scrub_nproc(ctx));
-		if (!ve.rvp_realtime) {
+		if (!vs.rvp_realtime) {
 			moveon = false;
 			str_info(ctx, ctx->mntpoint,
 	_("Could not create realtime device media verifier."));
 			goto out_logpool;
 		}
 	}
-	moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &ve);
+	moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &vs);
 	if (!moveon)
 		goto out_rtpool;
-	ctx->bytes_checked += clean_pool(ve.rvp_data);
-	ctx->bytes_checked += clean_pool(ve.rvp_log);
-	ctx->bytes_checked += clean_pool(ve.rvp_realtime);
+	ctx->bytes_checked += clean_pool(vs.rvp_data);
+	ctx->bytes_checked += clean_pool(vs.rvp_log);
+	ctx->bytes_checked += clean_pool(vs.rvp_realtime);
 
 	/* Scan the whole dir tree to see what matches the bad extents. */
-	if (!bitmap_empty(ve.d_bad) || !bitmap_empty(ve.r_bad))
-		moveon = xfs_report_verify_errors(ctx, ve.d_bad, ve.r_bad);
+	if (!bitmap_empty(vs.d_bad) || !bitmap_empty(vs.r_bad))
+		moveon = xfs_report_verify_errors(ctx, vs.d_bad, vs.r_bad);
 
-	bitmap_free(&ve.r_bad);
-	bitmap_free(&ve.d_bad);
+	bitmap_free(&vs.r_bad);
+	bitmap_free(&vs.d_bad);
 	return moveon;
 
 out_rtpool:
-	if (ve.rvp_realtime)
-		read_verify_pool_destroy(ve.rvp_realtime);
+	if (vs.rvp_realtime)
+		read_verify_pool_destroy(vs.rvp_realtime);
 out_logpool:
-	if (ve.rvp_log)
-		read_verify_pool_destroy(ve.rvp_log);
+	if (vs.rvp_log)
+		read_verify_pool_destroy(vs.rvp_log);
 out_datapool:
-	read_verify_pool_destroy(ve.rvp_data);
+	read_verify_pool_destroy(vs.rvp_data);
 out_rbad:
-	bitmap_free(&ve.r_bad);
+	bitmap_free(&vs.r_bad);
 out_dbad:
-	bitmap_free(&ve.d_bad);
+	bitmap_free(&vs.d_bad);
 out:
 	return moveon;
 }

Re: [PATCH v2 45/36] xfs_scrub: rename confusing structure

[Eric Sandeen]

On 4/4/19 12:10 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Rename xfs_verify_extent to media_verify_state so that the name more
> accurately reflects what we're storing in the structure.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/phase6.c |   70 ++++++++++++++++++++++++++++----------------------------
>  1 file changed, 35 insertions(+), 35 deletions(-)
> 
> diff --git a/scrub/phase6.c b/scrub/phase6.c
> index 5e2b9ee3..91e4443e 100644
> --- a/scrub/phase6.c
> +++ b/scrub/phase6.c
> @@ -35,7 +35,7 @@
>  
>  /* Verify disk blocks with GETFSMAP */
>  
> -struct xfs_verify_extent {
> +struct media_verify_state {
>  	struct read_verify_pool	*rvp_data;
>  	struct read_verify_pool	*rvp_log;
>  	struct read_verify_pool	*rvp_realtime;
> @@ -47,15 +47,15 @@ struct xfs_verify_extent {
>  static struct read_verify_pool *
>  xfs_dev_to_pool(
>  	struct scrub_ctx		*ctx,
> -	struct xfs_verify_extent	*ve,
> +	struct media_verify_state	*vs,
>  	dev_t				dev)
>  {
>  	if (dev == ctx->fsinfo.fs_datadev)
> -		return ve->rvp_data;
> +		return vs->rvp_data;
>  	else if (dev == ctx->fsinfo.fs_logdev)
> -		return ve->rvp_log;
> +		return vs->rvp_log;
>  	else if (dev == ctx->fsinfo.fs_rtdev)
> -		return ve->rvp_realtime;
> +		return vs->rvp_realtime;
>  	abort();
>  }
>  
> @@ -348,7 +348,7 @@ xfs_check_rmap_ioerr(
>  {
>  	struct fsmap			keys[2];
>  	char				descr[DESCR_BUFSZ];
> -	struct xfs_verify_extent	*ve = arg;
> +	struct media_verify_state	*vs = arg;
>  	struct bitmap			*tree;
>  	dev_t				dev;
>  	bool				moveon;
> @@ -360,9 +360,9 @@ xfs_check_rmap_ioerr(
>  	 * later rescanning.
>  	 */
>  	if (dev == ctx->fsinfo.fs_datadev)
> -		tree = ve->d_bad;
> +		tree = vs->d_bad;
>  	else if (dev == ctx->fsinfo.fs_rtdev)
> -		tree = ve->r_bad;
> +		tree = vs->r_bad;
>  	else
>  		tree = NULL;
>  	if (tree) {
> @@ -395,10 +395,10 @@ xfs_check_rmap(
>  	struct fsmap			*map,
>  	void				*arg)
>  {
> -	struct xfs_verify_extent	*ve = arg;
> +	struct media_verify_state	*vs = arg;
>  	struct read_verify_pool		*rvp;
>  
> -	rvp = xfs_dev_to_pool(ctx, ve, map->fmr_device);
> +	rvp = xfs_dev_to_pool(ctx, vs, map->fmr_device);
>  
>  	dbg_printf("rmap dev %d:%d phys %"PRIu64" owner %"PRId64
>  			" offset %"PRIu64" len %"PRIu64" flags 0x%x\n",
> @@ -425,7 +425,7 @@ xfs_check_rmap(
>  	/* XXX: Filter out directory data blocks. */
>  
>  	/* Schedule the read verify command for (eventual) running. */
> -	read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, ve);
> +	read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, vs);
>  
>  out:
>  	/* Is this the last extent?  Fire off the read. */
> @@ -463,35 +463,35 @@ bool
>  xfs_scan_blocks(
>  	struct scrub_ctx		*ctx)
>  {
> -	struct xfs_verify_extent	ve = { NULL };
> +	struct media_verify_state	vs = { NULL };
>  	bool				moveon;
>  
> -	moveon = bitmap_init(&ve.d_bad);
> +	moveon = bitmap_init(&vs.d_bad);
>  	if (!moveon) {
>  		str_errno(ctx, ctx->mntpoint);
>  		goto out;
>  	}
>  
> -	moveon = bitmap_init(&ve.r_bad);
> +	moveon = bitmap_init(&vs.r_bad);
>  	if (!moveon) {
>  		str_errno(ctx, ctx->mntpoint);
>  		goto out_dbad;
>  	}
>  
> -	ve.rvp_data = read_verify_pool_init(ctx, ctx->datadev,
> +	vs.rvp_data = read_verify_pool_init(ctx, ctx->datadev,
>  			ctx->geo.blocksize, xfs_check_rmap_ioerr,
>  			scrub_nproc(ctx));
> -	if (!ve.rvp_data) {
> +	if (!vs.rvp_data) {
>  		moveon = false;
>  		str_info(ctx, ctx->mntpoint,
>  _("Could not create data device media verifier."));
>  		goto out_rbad;
>  	}
>  	if (ctx->logdev) {
> -		ve.rvp_log = read_verify_pool_init(ctx, ctx->logdev,
> +		vs.rvp_log = read_verify_pool_init(ctx, ctx->logdev,
>  				ctx->geo.blocksize, xfs_check_rmap_ioerr,
>  				scrub_nproc(ctx));
> -		if (!ve.rvp_log) {
> +		if (!vs.rvp_log) {
>  			moveon = false;
>  			str_info(ctx, ctx->mntpoint,
>  	_("Could not create log device media verifier."));
> @@ -499,43 +499,43 @@ _("Could not create data device media verifier."));
>  		}
>  	}
>  	if (ctx->rtdev) {
> -		ve.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev,
> +		vs.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev,
>  				ctx->geo.blocksize, xfs_check_rmap_ioerr,
>  				scrub_nproc(ctx));
> -		if (!ve.rvp_realtime) {
> +		if (!vs.rvp_realtime) {
>  			moveon = false;
>  			str_info(ctx, ctx->mntpoint,
>  	_("Could not create realtime device media verifier."));
>  			goto out_logpool;
>  		}
>  	}
> -	moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &ve);
> +	moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &vs);
>  	if (!moveon)
>  		goto out_rtpool;
> -	ctx->bytes_checked += clean_pool(ve.rvp_data);
> -	ctx->bytes_checked += clean_pool(ve.rvp_log);
> -	ctx->bytes_checked += clean_pool(ve.rvp_realtime);
> +	ctx->bytes_checked += clean_pool(vs.rvp_data);
> +	ctx->bytes_checked += clean_pool(vs.rvp_log);
> +	ctx->bytes_checked += clean_pool(vs.rvp_realtime);
>  
>  	/* Scan the whole dir tree to see what matches the bad extents. */
> -	if (!bitmap_empty(ve.d_bad) || !bitmap_empty(ve.r_bad))
> -		moveon = xfs_report_verify_errors(ctx, ve.d_bad, ve.r_bad);
> +	if (!bitmap_empty(vs.d_bad) || !bitmap_empty(vs.r_bad))
> +		moveon = xfs_report_verify_errors(ctx, vs.d_bad, vs.r_bad);
>  
> -	bitmap_free(&ve.r_bad);
> -	bitmap_free(&ve.d_bad);
> +	bitmap_free(&vs.r_bad);
> +	bitmap_free(&vs.d_bad);
>  	return moveon;
>  
>  out_rtpool:
> -	if (ve.rvp_realtime)
> -		read_verify_pool_destroy(ve.rvp_realtime);
> +	if (vs.rvp_realtime)
> +		read_verify_pool_destroy(vs.rvp_realtime);
>  out_logpool:
> -	if (ve.rvp_log)
> -		read_verify_pool_destroy(ve.rvp_log);
> +	if (vs.rvp_log)
> +		read_verify_pool_destroy(vs.rvp_log);
>  out_datapool:
> -	read_verify_pool_destroy(ve.rvp_data);
> +	read_verify_pool_destroy(vs.rvp_data);
>  out_rbad:
> -	bitmap_free(&ve.r_bad);
> +	bitmap_free(&vs.r_bad);
>  out_dbad:
> -	bitmap_free(&ve.d_bad);
> +	bitmap_free(&vs.d_bad);
>  out:
>  	return moveon;
>  }
> 

[PATCH 46/36] xfs_scrub: remove pointless xfs_verify_error_info struct

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

The xfs_verify_error_info structure is a strict subset of
media_verify_info so just pass that around.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase6.c |   20 +++++---------------
 1 file changed, 5 insertions(+), 15 deletions(-)

diff --git a/scrub/phase6.c b/scrub/phase6.c
index 2b4f6ff1..f173f248 100644
--- a/scrub/phase6.c
+++ b/scrub/phase6.c
@@ -111,11 +111,6 @@ xfs_decode_special_owner(
 
 /* Routines to translate bad physical extents into file paths and offsets. */
 
-struct xfs_verify_error_info {
-	struct bitmap			*d_bad;		/* bytes */
-	struct bitmap			*r_bad;		/* bytes */
-};
-
 /* Report if this extent overlaps a bad region. */
 static bool
 xfs_report_verify_inode_bmap(
@@ -127,7 +122,7 @@ xfs_report_verify_inode_bmap(
 	struct xfs_bmap			*bmap,
 	void				*arg)
 {
-	struct xfs_verify_error_info	*vei = arg;
+	struct media_verify_state	*vei = arg;
 	struct bitmap			*bmp;
 
 	/* Only report errors for real extents. */
@@ -277,23 +272,18 @@ xfs_report_verify_dirent(
 static bool
 xfs_report_verify_errors(
 	struct scrub_ctx		*ctx,
-	struct bitmap			*d_bad,
-	struct bitmap			*r_bad)
+	struct media_verify_state	*ve)
 {
-	struct xfs_verify_error_info	vei;
 	bool				moveon;
 
-	vei.d_bad = d_bad;
-	vei.r_bad = r_bad;
-
 	/* Scan the directory tree to get file paths. */
 	moveon = scan_fs_tree(ctx, xfs_report_verify_dir,
-			xfs_report_verify_dirent, &vei);
+			xfs_report_verify_dirent, ve);
 	if (!moveon)
 		return false;
 
 	/* Scan for unlinked files. */
-	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
+	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, ve);
 }
 
 /* Report an IO error resulting from read-verify based off getfsmap. */
@@ -518,7 +508,7 @@ _("Could not create data device media verifier."));
 
 	/* Scan the whole dir tree to see what matches the bad extents. */
 	if (!bitmap_empty(ve.d_bad) || !bitmap_empty(ve.r_bad))
-		moveon = xfs_report_verify_errors(ctx, ve.d_bad, ve.r_bad);
+		moveon = xfs_report_verify_errors(ctx, &ve);
 
 	bitmap_free(&ve.r_bad);
 	bitmap_free(&ve.d_bad);

Re: [PATCH 46/36] xfs_scrub: remove pointless xfs_verify_error_info struct

[Eric Sandeen]

On 3/20/19 3:04 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> The xfs_verify_error_info structure is a strict subset of
> media_verify_info so just pass that around.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Uh, sorry but again, "*vei" isn't intuitive anymore...

> ---
>  scrub/phase6.c |   20 +++++---------------
>  1 file changed, 5 insertions(+), 15 deletions(-)
> 
> diff --git a/scrub/phase6.c b/scrub/phase6.c
> index 2b4f6ff1..f173f248 100644
> --- a/scrub/phase6.c
> +++ b/scrub/phase6.c
> @@ -111,11 +111,6 @@ xfs_decode_special_owner(
>  
>  /* Routines to translate bad physical extents into file paths and offsets. */
>  
> -struct xfs_verify_error_info {
> -	struct bitmap			*d_bad;		/* bytes */
> -	struct bitmap			*r_bad;		/* bytes */
> -};
> -
>  /* Report if this extent overlaps a bad region. */
>  static bool
>  xfs_report_verify_inode_bmap(
> @@ -127,7 +122,7 @@ xfs_report_verify_inode_bmap(
>  	struct xfs_bmap			*bmap,
>  	void				*arg)
>  {
> -	struct xfs_verify_error_info	*vei = arg;
> +	struct media_verify_state	*vei = arg;
>  	struct bitmap			*bmp;
>  
>  	/* Only report errors for real extents. */
> @@ -277,23 +272,18 @@ xfs_report_verify_dirent(
>  static bool
>  xfs_report_verify_errors(
>  	struct scrub_ctx		*ctx,
> -	struct bitmap			*d_bad,
> -	struct bitmap			*r_bad)
> +	struct media_verify_state	*ve)
>  {
> -	struct xfs_verify_error_info	vei;
>  	bool				moveon;
>  
> -	vei.d_bad = d_bad;
> -	vei.r_bad = r_bad;
> -
>  	/* Scan the directory tree to get file paths. */
>  	moveon = scan_fs_tree(ctx, xfs_report_verify_dir,
> -			xfs_report_verify_dirent, &vei);
> +			xfs_report_verify_dirent, ve);
>  	if (!moveon)
>  		return false;
>  
>  	/* Scan for unlinked files. */
> -	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
> +	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, ve);
>  }
>  
>  /* Report an IO error resulting from read-verify based off getfsmap. */
> @@ -518,7 +508,7 @@ _("Could not create data device media verifier."));
>  
>  	/* Scan the whole dir tree to see what matches the bad extents. */
>  	if (!bitmap_empty(ve.d_bad) || !bitmap_empty(ve.r_bad))
> -		moveon = xfs_report_verify_errors(ctx, ve.d_bad, ve.r_bad);
> +		moveon = xfs_report_verify_errors(ctx, &ve);
>  
>  	bitmap_free(&ve.r_bad);
>  	bitmap_free(&ve.d_bad);
> 

[PATCH v2 46/36] xfs_scrub: remove pointless xfs_verify_error_info struct

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

The xfs_verify_error_info structure is a strict subset of
media_verify_info so just pass that around.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase6.c |   24 +++++++-----------------
 1 file changed, 7 insertions(+), 17 deletions(-)

diff --git a/scrub/phase6.c b/scrub/phase6.c
index 91e4443e..e819e085 100644
--- a/scrub/phase6.c
+++ b/scrub/phase6.c
@@ -111,11 +111,6 @@ xfs_decode_special_owner(
 
 /* Routines to translate bad physical extents into file paths and offsets. */
 
-struct xfs_verify_error_info {
-	struct bitmap			*d_bad;		/* bytes */
-	struct bitmap			*r_bad;		/* bytes */
-};
-
 /* Report if this extent overlaps a bad region. */
 static bool
 xfs_report_verify_inode_bmap(
@@ -127,7 +122,7 @@ xfs_report_verify_inode_bmap(
 	struct xfs_bmap			*bmap,
 	void				*arg)
 {
-	struct xfs_verify_error_info	*vei = arg;
+	struct media_verify_state	*vs = arg;
 	struct bitmap			*bmp;
 
 	/* Only report errors for real extents. */
@@ -135,9 +130,9 @@ xfs_report_verify_inode_bmap(
 		return true;
 
 	if (fsx->fsx_xflags & FS_XFLAG_REALTIME)
-		bmp = vei->r_bad;
+		bmp = vs->r_bad;
 	else
-		bmp = vei->d_bad;
+		bmp = vs->d_bad;
 
 	if (!bitmap_test(bmp, bmap->bm_physical, bmap->bm_length))
 		return true;
@@ -277,23 +272,18 @@ xfs_report_verify_dirent(
 static bool
 xfs_report_verify_errors(
 	struct scrub_ctx		*ctx,
-	struct bitmap			*d_bad,
-	struct bitmap			*r_bad)
+	struct media_verify_state	*ve)
 {
-	struct xfs_verify_error_info	vei;
 	bool				moveon;
 
-	vei.d_bad = d_bad;
-	vei.r_bad = r_bad;
-
 	/* Scan the directory tree to get file paths. */
 	moveon = scan_fs_tree(ctx, xfs_report_verify_dir,
-			xfs_report_verify_dirent, &vei);
+			xfs_report_verify_dirent, ve);
 	if (!moveon)
 		return false;
 
 	/* Scan for unlinked files. */
-	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
+	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, ve);
 }
 
 /* Report an IO error resulting from read-verify based off getfsmap. */
@@ -518,7 +508,7 @@ _("Could not create data device media verifier."));
 
 	/* Scan the whole dir tree to see what matches the bad extents. */
 	if (!bitmap_empty(vs.d_bad) || !bitmap_empty(vs.r_bad))
-		moveon = xfs_report_verify_errors(ctx, vs.d_bad, vs.r_bad);
+		moveon = xfs_report_verify_errors(ctx, &vs);
 
 	bitmap_free(&vs.r_bad);
 	bitmap_free(&vs.d_bad);

Re: [PATCH v2 46/36] xfs_scrub: remove pointless xfs_verify_error_info struct

[Eric Sandeen]

On 4/4/19 12:10 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> The xfs_verify_error_info structure is a strict subset of
> media_verify_info so just pass that around.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

imagonna change 

> +	struct media_verify_state	*ve)

to

> +	struct media_verify_state	*vs)

as well on commit, and if that's ok with you, 

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/phase6.c |   24 +++++++-----------------
>  1 file changed, 7 insertions(+), 17 deletions(-)
> 
> diff --git a/scrub/phase6.c b/scrub/phase6.c
> index 91e4443e..e819e085 100644
> --- a/scrub/phase6.c
> +++ b/scrub/phase6.c
> @@ -111,11 +111,6 @@ xfs_decode_special_owner(
>  
>  /* Routines to translate bad physical extents into file paths and offsets. */
>  
> -struct xfs_verify_error_info {
> -	struct bitmap			*d_bad;		/* bytes */
> -	struct bitmap			*r_bad;		/* bytes */
> -};
> -
>  /* Report if this extent overlaps a bad region. */
>  static bool
>  xfs_report_verify_inode_bmap(
> @@ -127,7 +122,7 @@ xfs_report_verify_inode_bmap(
>  	struct xfs_bmap			*bmap,
>  	void				*arg)
>  {
> -	struct xfs_verify_error_info	*vei = arg;
> +	struct media_verify_state	*vs = arg;
>  	struct bitmap			*bmp;
>  
>  	/* Only report errors for real extents. */
> @@ -135,9 +130,9 @@ xfs_report_verify_inode_bmap(
>  		return true;
>  
>  	if (fsx->fsx_xflags & FS_XFLAG_REALTIME)
> -		bmp = vei->r_bad;
> +		bmp = vs->r_bad;
>  	else
> -		bmp = vei->d_bad;
> +		bmp = vs->d_bad;
>  
>  	if (!bitmap_test(bmp, bmap->bm_physical, bmap->bm_length))
>  		return true;
> @@ -277,23 +272,18 @@ xfs_report_verify_dirent(
>  static bool
>  xfs_report_verify_errors(
>  	struct scrub_ctx		*ctx,
> -	struct bitmap			*d_bad,
> -	struct bitmap			*r_bad)
> +	struct media_verify_state	*ve)
>  {
> -	struct xfs_verify_error_info	vei;
>  	bool				moveon;
>  
> -	vei.d_bad = d_bad;
> -	vei.r_bad = r_bad;
> -
>  	/* Scan the directory tree to get file paths. */
>  	moveon = scan_fs_tree(ctx, xfs_report_verify_dir,
> -			xfs_report_verify_dirent, &vei);
> +			xfs_report_verify_dirent, ve);
>  	if (!moveon)
>  		return false;
>  
>  	/* Scan for unlinked files. */
> -	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
> +	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, ve);
>  }
>  
>  /* Report an IO error resulting from read-verify based off getfsmap. */
> @@ -518,7 +508,7 @@ _("Could not create data device media verifier."));
>  
>  	/* Scan the whole dir tree to see what matches the bad extents. */
>  	if (!bitmap_empty(vs.d_bad) || !bitmap_empty(vs.r_bad))
> -		moveon = xfs_report_verify_errors(ctx, vs.d_bad, vs.r_bad);
> +		moveon = xfs_report_verify_errors(ctx, &vs);
>  
>  	bitmap_free(&vs.r_bad);
>  	bitmap_free(&vs.d_bad);
> 

Re: [PATCH v2 46/36] xfs_scrub: remove pointless xfs_verify_error_info struct

[Darrick J. Wong]

On Fri, Apr 12, 2019 at 02:23:59PM -0500, Eric Sandeen wrote:
> On 4/4/19 12:10 PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > The xfs_verify_error_info structure is a strict subset of
> > media_verify_info so just pass that around.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> imagonna change 
> 
> > +	struct media_verify_state	*ve)
> 
> to
> 
> > +	struct media_verify_state	*vs)
> 
> as well on commit, and if that's ok with you, 

Yep, fine with me.

--D

> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
> 
> > ---
> >  scrub/phase6.c |   24 +++++++-----------------
> >  1 file changed, 7 insertions(+), 17 deletions(-)
> > 
> > diff --git a/scrub/phase6.c b/scrub/phase6.c
> > index 91e4443e..e819e085 100644
> > --- a/scrub/phase6.c
> > +++ b/scrub/phase6.c
> > @@ -111,11 +111,6 @@ xfs_decode_special_owner(
> >  
> >  /* Routines to translate bad physical extents into file paths and offsets. */
> >  
> > -struct xfs_verify_error_info {
> > -	struct bitmap			*d_bad;		/* bytes */
> > -	struct bitmap			*r_bad;		/* bytes */
> > -};
> > -
> >  /* Report if this extent overlaps a bad region. */
> >  static bool
> >  xfs_report_verify_inode_bmap(
> > @@ -127,7 +122,7 @@ xfs_report_verify_inode_bmap(
> >  	struct xfs_bmap			*bmap,
> >  	void				*arg)
> >  {
> > -	struct xfs_verify_error_info	*vei = arg;
> > +	struct media_verify_state	*vs = arg;
> >  	struct bitmap			*bmp;
> >  
> >  	/* Only report errors for real extents. */
> > @@ -135,9 +130,9 @@ xfs_report_verify_inode_bmap(
> >  		return true;
> >  
> >  	if (fsx->fsx_xflags & FS_XFLAG_REALTIME)
> > -		bmp = vei->r_bad;
> > +		bmp = vs->r_bad;
> >  	else
> > -		bmp = vei->d_bad;
> > +		bmp = vs->d_bad;
> >  
> >  	if (!bitmap_test(bmp, bmap->bm_physical, bmap->bm_length))
> >  		return true;
> > @@ -277,23 +272,18 @@ xfs_report_verify_dirent(
> >  static bool
> >  xfs_report_verify_errors(
> >  	struct scrub_ctx		*ctx,
> > -	struct bitmap			*d_bad,
> > -	struct bitmap			*r_bad)
> > +	struct media_verify_state	*ve)
> >  {
> > -	struct xfs_verify_error_info	vei;
> >  	bool				moveon;
> >  
> > -	vei.d_bad = d_bad;
> > -	vei.r_bad = r_bad;
> > -
> >  	/* Scan the directory tree to get file paths. */
> >  	moveon = scan_fs_tree(ctx, xfs_report_verify_dir,
> > -			xfs_report_verify_dirent, &vei);
> > +			xfs_report_verify_dirent, ve);
> >  	if (!moveon)
> >  		return false;
> >  
> >  	/* Scan for unlinked files. */
> > -	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, &vei);
> > +	return xfs_scan_all_inodes(ctx, xfs_report_verify_inode, ve);
> >  }
> >  
> >  /* Report an IO error resulting from read-verify based off getfsmap. */
> > @@ -518,7 +508,7 @@ _("Could not create data device media verifier."));
> >  
> >  	/* Scan the whole dir tree to see what matches the bad extents. */
> >  	if (!bitmap_empty(vs.d_bad) || !bitmap_empty(vs.r_bad))
> > -		moveon = xfs_report_verify_errors(ctx, vs.d_bad, vs.r_bad);
> > +		moveon = xfs_report_verify_errors(ctx, &vs);
> >  
> >  	bitmap_free(&vs.r_bad);
> >  	bitmap_free(&vs.d_bad);
> > 

[PATCH 47/36] xfs_scrub: remove xfs_ prefixes from structure names

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Drop the xfs_ prefix from scrub's struct definitions since they're
private to the program.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/phase5.c |    8 +++----
 scrub/phase7.c |   66 ++++++++++++++++++++++++++++----------------------------
 2 files changed, 37 insertions(+), 37 deletions(-)

diff --git a/scrub/phase5.c b/scrub/phase5.c
index 49886e6f..1743119d 100644
--- a/scrub/phase5.c
+++ b/scrub/phase5.c
@@ -119,12 +119,12 @@ xfs_scrub_scan_dirents(
 
 #ifdef HAVE_LIBATTR
 /* Routines to scan all of an inode's xattrs for name problems. */
-struct xfs_attr_ns {
+struct attrns_decode {
 	int			flags;
 	const char		*name;
 };
 
-static const struct xfs_attr_ns attr_ns[] = {
+static const struct attrns_decode attr_ns[] = {
 	{0,			"user"},
 	{ATTR_ROOT,		"system"},
 	{ATTR_SECURE,		"secure"},
@@ -141,7 +141,7 @@ xfs_scrub_scan_fhandle_namespace_xattrs(
 	const char			*descr,
 	struct xfs_handle		*handle,
 	struct xfs_bstat		*bstat,
-	const struct xfs_attr_ns	*attr_ns)
+	const struct attrns_decode	*attr_ns)
 {
 	struct attrlist_cursor		cur;
 	char				attrbuf[XFS_XATTR_LIST_MAX];
@@ -202,7 +202,7 @@ xfs_scrub_scan_fhandle_xattrs(
 	struct xfs_handle		*handle,
 	struct xfs_bstat		*bstat)
 {
-	const struct xfs_attr_ns	*ns;
+	const struct attrns_decode	*ns;
 	bool				moveon = true;
 
 	for (ns = attr_ns; ns->name; ns++) {
diff --git a/scrub/phase7.c b/scrub/phase7.c
index 504a6927..0c3202e4 100644
--- a/scrub/phase7.c
+++ b/scrub/phase7.c
@@ -16,7 +16,7 @@
 
 /* Phase 7: Check summary counters. */
 
-struct xfs_summary_counts {
+struct summary_counts {
 	unsigned long long	dbytes;		/* data dev bytes */
 	unsigned long long	rbytes;		/* rt dev bytes */
 	unsigned long long	next_phys;	/* next phys bytes we see? */
@@ -26,13 +26,13 @@ struct xfs_summary_counts {
 /* Record block usage. */
 static bool
 xfs_record_block_summary(
-	struct scrub_ctx		*ctx,
-	const char			*descr,
-	struct fsmap			*fsmap,
-	void				*arg)
+	struct scrub_ctx	*ctx,
+	const char		*descr,
+	struct fsmap		*fsmap,
+	void			*arg)
 {
-	struct xfs_summary_counts	*counts;
-	unsigned long long		len;
+	struct summary_counts	*counts;
+	unsigned long long	len;
 
 	counts = ptvar_get((struct ptvar *)arg);
 	if (fsmap->fmr_device == ctx->fsinfo.fs_logdev)
@@ -67,12 +67,12 @@ xfs_record_block_summary(
 /* Add all the summaries in the per-thread counter */
 static bool
 xfs_add_summaries(
-	struct ptvar			*ptv,
-	void				*data,
-	void				*arg)
+	struct ptvar		*ptv,
+	void			*data,
+	void			*arg)
 {
-	struct xfs_summary_counts	*total = arg;
-	struct xfs_summary_counts	*item = data;
+	struct summary_counts	*total = arg;
+	struct summary_counts	*item = data;
 
 	total->dbytes += item->dbytes;
 	total->rbytes += item->rbytes;
@@ -88,27 +88,27 @@ xfs_add_summaries(
  */
 bool
 xfs_scan_summary(
-	struct scrub_ctx		*ctx)
+	struct scrub_ctx	*ctx)
 {
-	struct xfs_summary_counts	totalcount = {0};
-	struct ptvar			*ptvar;
-	unsigned long long		used_data;
-	unsigned long long		used_rt;
-	unsigned long long		used_files;
-	unsigned long long		stat_data;
-	unsigned long long		stat_rt;
-	uint64_t			counted_inodes = 0;
-	unsigned long long		absdiff;
-	unsigned long long		d_blocks;
-	unsigned long long		d_bfree;
-	unsigned long long		r_blocks;
-	unsigned long long		r_bfree;
-	unsigned long long		f_files;
-	unsigned long long		f_free;
-	bool				moveon;
-	bool				complain;
-	int				ip;
-	int				error;
+	struct summary_counts	totalcount = {0};
+	struct ptvar		*ptvar;
+	unsigned long long	used_data;
+	unsigned long long	used_rt;
+	unsigned long long	used_files;
+	unsigned long long	stat_data;
+	unsigned long long	stat_rt;
+	uint64_t		counted_inodes = 0;
+	unsigned long long	absdiff;
+	unsigned long long	d_blocks;
+	unsigned long long	d_bfree;
+	unsigned long long	r_blocks;
+	unsigned long long	r_bfree;
+	unsigned long long	f_files;
+	unsigned long long	f_free;
+	bool			moveon;
+	bool			complain;
+	int			ip;
+	int			error;
 
 	/* Flush everything out to disk before we start counting. */
 	error = syncfs(ctx->mnt_fd);
@@ -117,7 +117,7 @@ xfs_scan_summary(
 		return false;
 	}
 
-	ptvar = ptvar_init(scrub_nproc(ctx), sizeof(struct xfs_summary_counts));
+	ptvar = ptvar_init(scrub_nproc(ctx), sizeof(struct summary_counts));
 	if (!ptvar) {
 		str_errno(ctx, ctx->mntpoint);
 		return false;

Re: [PATCH 47/36] xfs_scrub: remove xfs_ prefixes from structure names

[Eric Sandeen]

On 3/20/19 3:05 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Drop the xfs_ prefix from scrub's struct definitions since they're
> private to the program.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/phase5.c |    8 +++----
>  scrub/phase7.c |   66 ++++++++++++++++++++++++++++----------------------------
>  2 files changed, 37 insertions(+), 37 deletions(-)
> 
> diff --git a/scrub/phase5.c b/scrub/phase5.c
> index 49886e6f..1743119d 100644
> --- a/scrub/phase5.c
> +++ b/scrub/phase5.c
> @@ -119,12 +119,12 @@ xfs_scrub_scan_dirents(
>  
>  #ifdef HAVE_LIBATTR
>  /* Routines to scan all of an inode's xattrs for name problems. */
> -struct xfs_attr_ns {
> +struct attrns_decode {
>  	int			flags;
>  	const char		*name;
>  };
>  
> -static const struct xfs_attr_ns attr_ns[] = {
> +static const struct attrns_decode attr_ns[] = {
>  	{0,			"user"},
>  	{ATTR_ROOT,		"system"},
>  	{ATTR_SECURE,		"secure"},
> @@ -141,7 +141,7 @@ xfs_scrub_scan_fhandle_namespace_xattrs(
>  	const char			*descr,
>  	struct xfs_handle		*handle,
>  	struct xfs_bstat		*bstat,
> -	const struct xfs_attr_ns	*attr_ns)
> +	const struct attrns_decode	*attr_ns)
>  {
>  	struct attrlist_cursor		cur;
>  	char				attrbuf[XFS_XATTR_LIST_MAX];
> @@ -202,7 +202,7 @@ xfs_scrub_scan_fhandle_xattrs(
>  	struct xfs_handle		*handle,
>  	struct xfs_bstat		*bstat)
>  {
> -	const struct xfs_attr_ns	*ns;
> +	const struct attrns_decode	*ns;
>  	bool				moveon = true;
>  
>  	for (ns = attr_ns; ns->name; ns++) {
> diff --git a/scrub/phase7.c b/scrub/phase7.c
> index 504a6927..0c3202e4 100644
> --- a/scrub/phase7.c
> +++ b/scrub/phase7.c
> @@ -16,7 +16,7 @@
>  
>  /* Phase 7: Check summary counters. */
>  
> -struct xfs_summary_counts {
> +struct summary_counts {
>  	unsigned long long	dbytes;		/* data dev bytes */
>  	unsigned long long	rbytes;		/* rt dev bytes */
>  	unsigned long long	next_phys;	/* next phys bytes we see? */
> @@ -26,13 +26,13 @@ struct xfs_summary_counts {
>  /* Record block usage. */
>  static bool
>  xfs_record_block_summary(
> -	struct scrub_ctx		*ctx,
> -	const char			*descr,
> -	struct fsmap			*fsmap,
> -	void				*arg)
> +	struct scrub_ctx	*ctx,
> +	const char		*descr,
> +	struct fsmap		*fsmap,
> +	void			*arg)
>  {
> -	struct xfs_summary_counts	*counts;
> -	unsigned long long		len;
> +	struct summary_counts	*counts;
> +	unsigned long long	len;
>  
>  	counts = ptvar_get((struct ptvar *)arg);
>  	if (fsmap->fmr_device == ctx->fsinfo.fs_logdev)
> @@ -67,12 +67,12 @@ xfs_record_block_summary(
>  /* Add all the summaries in the per-thread counter */
>  static bool
>  xfs_add_summaries(
> -	struct ptvar			*ptv,
> -	void				*data,
> -	void				*arg)
> +	struct ptvar		*ptv,
> +	void			*data,
> +	void			*arg)
>  {
> -	struct xfs_summary_counts	*total = arg;
> -	struct xfs_summary_counts	*item = data;
> +	struct summary_counts	*total = arg;
> +	struct summary_counts	*item = data;
>  
>  	total->dbytes += item->dbytes;
>  	total->rbytes += item->rbytes;
> @@ -88,27 +88,27 @@ xfs_add_summaries(
>   */
>  bool
>  xfs_scan_summary(
> -	struct scrub_ctx		*ctx)
> +	struct scrub_ctx	*ctx)
>  {
> -	struct xfs_summary_counts	totalcount = {0};
> -	struct ptvar			*ptvar;
> -	unsigned long long		used_data;
> -	unsigned long long		used_rt;
> -	unsigned long long		used_files;
> -	unsigned long long		stat_data;
> -	unsigned long long		stat_rt;
> -	uint64_t			counted_inodes = 0;
> -	unsigned long long		absdiff;
> -	unsigned long long		d_blocks;
> -	unsigned long long		d_bfree;
> -	unsigned long long		r_blocks;
> -	unsigned long long		r_bfree;
> -	unsigned long long		f_files;
> -	unsigned long long		f_free;
> -	bool				moveon;
> -	bool				complain;
> -	int				ip;
> -	int				error;
> +	struct summary_counts	totalcount = {0};
> +	struct ptvar		*ptvar;
> +	unsigned long long	used_data;
> +	unsigned long long	used_rt;
> +	unsigned long long	used_files;
> +	unsigned long long	stat_data;
> +	unsigned long long	stat_rt;
> +	uint64_t		counted_inodes = 0;
> +	unsigned long long	absdiff;
> +	unsigned long long	d_blocks;
> +	unsigned long long	d_bfree;
> +	unsigned long long	r_blocks;
> +	unsigned long long	r_bfree;
> +	unsigned long long	f_files;
> +	unsigned long long	f_free;
> +	bool			moveon;
> +	bool			complain;
> +	int			ip;
> +	int			error;
>  
>  	/* Flush everything out to disk before we start counting. */
>  	error = syncfs(ctx->mnt_fd);
> @@ -117,7 +117,7 @@ xfs_scan_summary(
>  		return false;
>  	}
>  
> -	ptvar = ptvar_init(scrub_nproc(ctx), sizeof(struct xfs_summary_counts));
> +	ptvar = ptvar_init(scrub_nproc(ctx), sizeof(struct summary_counts));
>  	if (!ptvar) {
>  		str_errno(ctx, ctx->mntpoint);
>  		return false;
> 

[PATCH 48/36] mkfs: don't use DIFLAG values for fsx_xflags

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

mkfs shouldn't be using DIFLAG values where FS_XFLAG values are
required.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 mkfs/xfs_mkfs.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
index e87c692c..19ad1c90 100644
--- a/mkfs/xfs_mkfs.c
+++ b/mkfs/xfs_mkfs.c
@@ -1451,15 +1451,15 @@ data_opts_parser(
 		break;
 	case D_RTINHERIT:
 		if (getnum(value, opts, subopt))
-			cli->fsx.fsx_xflags |= XFS_DIFLAG_RTINHERIT;
+			cli->fsx.fsx_xflags |= FS_XFLAG_RTINHERIT;
 		break;
 	case D_PROJINHERIT:
 		cli->fsx.fsx_projid = getnum(value, opts, subopt);
-		cli->fsx.fsx_xflags |= XFS_DIFLAG_PROJINHERIT;
+		cli->fsx.fsx_xflags |= FS_XFLAG_PROJINHERIT;
 		break;
 	case D_EXTSZINHERIT:
 		cli->fsx.fsx_extsize = getnum(value, opts, subopt);
-		cli->fsx.fsx_xflags |= XFS_DIFLAG_EXTSZINHERIT;
+		cli->fsx.fsx_xflags |= FS_XFLAG_EXTSZINHERIT;
 		break;
 	case D_COWEXTSIZE:
 		cli->fsx.fsx_cowextsize = getnum(value, opts, subopt);
@@ -2212,9 +2212,9 @@ validate_extsize_hint(
 	bool			rt;
 	uint16_t		flags = 0;
 
-	rt = (cli->fsx.fsx_xflags & XFS_DIFLAG_RTINHERIT);
+	rt = (cli->fsx.fsx_xflags & FS_XFLAG_RTINHERIT);
 
-	if (cli->fsx.fsx_xflags & XFS_DIFLAG_EXTSZINHERIT)
+	if (cli->fsx.fsx_xflags & FS_XFLAG_EXTSZINHERIT)
 		flags |= XFS_DIFLAG_EXTSIZE;
 
 	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFREG,

Re: [PATCH 48/36] mkfs: don't use DIFLAG values for fsx_xflags

[Eric Sandeen]

On 3/26/19 12:18 PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> mkfs shouldn't be using DIFLAG values where FS_XFLAG values are
> required.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Yay!

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  mkfs/xfs_mkfs.c |   10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
> index e87c692c..19ad1c90 100644
> --- a/mkfs/xfs_mkfs.c
> +++ b/mkfs/xfs_mkfs.c
> @@ -1451,15 +1451,15 @@ data_opts_parser(
>  		break;
>  	case D_RTINHERIT:
>  		if (getnum(value, opts, subopt))
> -			cli->fsx.fsx_xflags |= XFS_DIFLAG_RTINHERIT;
> +			cli->fsx.fsx_xflags |= FS_XFLAG_RTINHERIT;
>  		break;
>  	case D_PROJINHERIT:
>  		cli->fsx.fsx_projid = getnum(value, opts, subopt);
> -		cli->fsx.fsx_xflags |= XFS_DIFLAG_PROJINHERIT;
> +		cli->fsx.fsx_xflags |= FS_XFLAG_PROJINHERIT;
>  		break;
>  	case D_EXTSZINHERIT:
>  		cli->fsx.fsx_extsize = getnum(value, opts, subopt);
> -		cli->fsx.fsx_xflags |= XFS_DIFLAG_EXTSZINHERIT;
> +		cli->fsx.fsx_xflags |= FS_XFLAG_EXTSZINHERIT;
>  		break;
>  	case D_COWEXTSIZE:
>  		cli->fsx.fsx_cowextsize = getnum(value, opts, subopt);
> @@ -2212,9 +2212,9 @@ validate_extsize_hint(
>  	bool			rt;
>  	uint16_t		flags = 0;
>  
> -	rt = (cli->fsx.fsx_xflags & XFS_DIFLAG_RTINHERIT);
> +	rt = (cli->fsx.fsx_xflags & FS_XFLAG_RTINHERIT);
>  
> -	if (cli->fsx.fsx_xflags & XFS_DIFLAG_EXTSZINHERIT)
> +	if (cli->fsx.fsx_xflags & FS_XFLAG_EXTSZINHERIT)
>  		flags |= XFS_DIFLAG_EXTSIZE;
>  
>  	fa = libxfs_inode_validate_extsize(mp, cli->fsx.fsx_extsize, S_IFREG,
> 

[PATCH 49/36] xfs_scrub: remove redundant function declarations

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Remove a redundant function declaration.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 scrub/scrub.h |    2 --
 1 file changed, 2 deletions(-)

diff --git a/scrub/scrub.h b/scrub/scrub.h
index 34c3dd30..e6e3f16f 100644
--- a/scrub/scrub.h
+++ b/scrub/scrub.h
@@ -25,8 +25,6 @@ bool xfs_scrub_ag_metadata(struct scrub_ctx *ctx, xfs_agnumber_t agno,
 		struct xfs_action_list *alist);
 bool xfs_scrub_fs_metadata(struct scrub_ctx *ctx,
 		struct xfs_action_list *alist);
-enum check_outcome xfs_repair_metadata(struct scrub_ctx *ctx, int fd,
-		struct action_item *aitem, unsigned int flags);
 
 bool xfs_can_scrub_fs_metadata(struct scrub_ctx *ctx);
 bool xfs_can_scrub_inode(struct scrub_ctx *ctx);

Re: [PATCH 49/36] xfs_scrub: remove redundant function declarations

[Eric Sandeen]

On 4/1/19 11:43 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Remove a redundant function declaration.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  scrub/scrub.h |    2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/scrub/scrub.h b/scrub/scrub.h
> index 34c3dd30..e6e3f16f 100644
> --- a/scrub/scrub.h
> +++ b/scrub/scrub.h
> @@ -25,8 +25,6 @@ bool xfs_scrub_ag_metadata(struct scrub_ctx *ctx, xfs_agnumber_t agno,
>  		struct xfs_action_list *alist);
>  bool xfs_scrub_fs_metadata(struct scrub_ctx *ctx,
>  		struct xfs_action_list *alist);
> -enum check_outcome xfs_repair_metadata(struct scrub_ctx *ctx, int fd,
> -		struct action_item *aitem, unsigned int flags);
>  
>  bool xfs_can_scrub_fs_metadata(struct scrub_ctx *ctx);
>  bool xfs_can_scrub_inode(struct scrub_ctx *ctx);
>