[PATCH v3 0/4] miscellaneous ZBD improvements

[PATCH v3 0/4] miscellaneous ZBD improvements

[Dmitry Fomichev]

This series contains four patches related to zoned block devices.

v2 -> v3:
  - initialized libzbc ioengine feature flag in configure
  - improved error output in check_min_lib_version()

v1 -> v2:
  - removed recursive zone locking in zbd_reset_zone()
  - added --disable-libzbc configure option
  - moved pkg-config version check to a separate function
  - added a patch to verify that pkg-config is available to check
    the minimum supported version in all libraries that need that

Dmitry Fomichev (4):
  configure: improve libzbc version check
  configure: check if pkg-config is installed
  zbd: simplify zone reset code
  t/zbd: check log file for failed assertions

 configure              | 62 +++++++++++++++++++---------------
 t/zbd/test-zbd-support |  9 ++++-
 zbd.c                  | 76 +++++++++++++++---------------------------
 3 files changed, 70 insertions(+), 77 deletions(-)

-- 
2.21.0

[PATCH v3 1/4] configure: improve libzbc version check

[Dmitry Fomichev]

Avoid parsing pkg-config output and just use --atleast-version to
check if libzbc is present and has an up to date version.

Currently, support for libzbc ioengine is always included if libzbc is
found at the build system. Add the option to disable libzbc ioengine
support even if libzbc is found. This can be useful for
cross-compilation.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
---
 configure | 29 ++++++++++++++++-------------
 1 file changed, 16 insertions(+), 13 deletions(-)

diff --git a/configure b/configure
index 5925e94f..81fd32bb 100755
--- a/configure
+++ b/configure
@@ -152,6 +152,7 @@ march_set="no"
 libiscsi="no"
 libnbd="no"
 libaio_uring="no"
+libzbc=""
 dynamic_engines="no"
 prefix=/usr/local
 
@@ -213,6 +214,8 @@ for opt do
   ;;
   --enable-libnbd) libnbd="yes"
   ;;
+  --disable-libzbc) libzbc="no"
+  ;;
   --disable-tcmalloc) disable_tcmalloc="yes"
   ;;
   --enable-libaio-uring) libaio_uring="yes"
@@ -256,6 +259,7 @@ if test "$show_help" = "yes" ; then
   echo "--with-ime=             Install path for DDN's Infinite Memory Engine"
   echo "--enable-libiscsi       Enable iscsi support"
   echo "--enable-libnbd         Enable libnbd (NBD engine) support"
+  echo "--disable-libzbc        Disable libzbc even if found"
   echo "--disable-tcmalloc	Disable tcmalloc support"
   echo "--enable-libaio-uring   Enable libaio emulated over io_uring"
   echo "--dynamic-libengines	Lib-based ioengines as dynamic libraries"
@@ -2454,9 +2458,6 @@ fi
 
 ##########################################
 # libzbc probe
-if test "$libzbc" != "yes" ; then
-  libzbc="no"
-fi
 cat > $TMPC << EOF
 #include <libzbc/zbc.h>
 int main(int argc, char **argv)
@@ -2466,19 +2467,21 @@ int main(int argc, char **argv)
   return zbc_open("foo=bar", O_RDONLY, &dev);
 }
 EOF
-if compile_prog "" "-lzbc" "libzbc"; then
-  libzbcvermaj=$(pkg-config --modversion libzbc | sed 's/\.[0-9]*\.[0-9]*//')
-  if test "$libzbcvermaj" -ge "5" ; then
-    libzbc="yes"
+if test "$libzbc" != "no" ; then
+  if compile_prog "" "-lzbc" "libzbc"; then
+    minimum_libzbc=5
+    if $(pkg-config --atleast-version=$minimum_libzbc libzbc); then
+      libzbc="yes"
+    else
+      print_config "libzbc engine" "libzbc version $minimum_libzbc or above required"
+      libzbc="no"
+    fi
   else
-    print_config "libzbc engine" "Unsupported libzbc version (version 5 or above required)"
+    if test "$libzbc" = "yes" ; then
+      feature_not_found "libzbc" "libzbc or libzbc/zbc.h"
+    fi
     libzbc="no"
   fi
-else
-  if test "$libzbc" = "yes" ; then
-      feature_not_found "libzbc" "libzbc or libzbc/zbc.h"
-  fi
-  libzbc="no"
 fi
 print_config "libzbc engine" "$libzbc"
 
-- 
2.21.0

[PATCH v3 2/4] configure: check if pkg-config is installed

[Dmitry Fomichev]

A few libraries need to be newer than a specific version in order to be
supported by fio and pkg-config utility is used to verify the versions
of the installed libraries. Since this step may fail because pkg-config
is not installed, verify pkg-config presence and warn the user if it
could not be found.

To avoid code duplication, add a common helper function to perform
these checks.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
---
 configure | 45 +++++++++++++++++++++++++--------------------
 1 file changed, 25 insertions(+), 20 deletions(-)

diff --git a/configure b/configure
index 81fd32bb..dd7fe3d2 100755
--- a/configure
+++ b/configure
@@ -133,6 +133,23 @@ output_sym() {
   echo "#define $1" >> $config_host_h
 }
 
+check_min_lib_version() {
+  local feature=$3
+
+  if ${cross_prefix}pkg-config --atleast-version=$2 $1 > /dev/null 2>&1; then
+    return 0
+  fi
+  : ${feature:=${1}}
+  if ${cross_prefix}pkg-config --version > /dev/null 2>&1; then
+    if test ${!feature} = "yes" ; then
+      feature_not_found "$feature" "$1 >= $2"
+    fi
+  else
+    print_config "$1" "missing pkg-config, can't check $feature version"
+  fi
+  return 1
+}
+
 targetos=""
 cpu=""
 
@@ -1521,18 +1538,17 @@ if test "$?" != "0" ; then
   echo "configure: gtk and gthread not found"
   exit 1
 fi
-if ! ${cross_prefix}pkg-config --atleast-version 2.18.0 gtk+-2.0; then
-  echo "GTK found, but need version 2.18 or higher"
-  gfio="no"
-else
+gfio="yes"
+if check_min_lib_version gtk+-2.0 2.18.0 "gfio"; then
   if compile_prog "$GTK_CFLAGS" "$GTK_LIBS" "gfio" ; then
-    gfio="yes"
     GFIO_LIBS="$LIBS $GTK_LIBS"
     CFLAGS="$CFLAGS $GTK_CFLAGS"
   else
     echo "Please install gtk and gdk libraries"
     gfio="no"
   fi
+else
+  gfio="no"
 fi
 LDFLAGS=$ORG_LDFLAGS
 fi
@@ -2182,15 +2198,11 @@ print_config "DDN's Infinite Memory Engine" "$libime"
 ##########################################
 # Check if we have libiscsi
 if test "$libiscsi" != "no" ; then
-  minimum_libiscsi=1.9.0
-  if $(pkg-config --atleast-version=$minimum_libiscsi libiscsi); then
+  if check_min_lib_version libiscsi 1.9.0; then
     libiscsi="yes"
     libiscsi_cflags=$(pkg-config --cflags libiscsi)
     libiscsi_libs=$(pkg-config --libs libiscsi)
   else
-    if test "$libiscsi" = "yes" ; then
-      feature_not_found "libiscsi" "libiscsi >= $minimum_libiscsi"
-    fi
     libiscsi="no"
   fi
 fi
@@ -2199,15 +2211,11 @@ print_config "iscsi engine" "$libiscsi"
 ##########################################
 # Check if we have libnbd (for NBD support)
 if test "$libnbd" != "no" ; then
-  minimum_libnbd=0.9.8
-  if $(pkg-config --atleast-version=$minimum_libnbd libnbd); then
+  if check_min_lib_version libnbd 0.9.8; then
     libnbd="yes"
     libnbd_cflags=$(pkg-config --cflags libnbd)
     libnbd_libs=$(pkg-config --libs libnbd)
   else
-    if test "$libnbd" = "yes" ; then
-      feature_not_found "libnbd" "libnbd >= $minimum_libnbd"
-    fi
     libnbd="no"
   fi
 fi
@@ -2469,11 +2477,8 @@ int main(int argc, char **argv)
 EOF
 if test "$libzbc" != "no" ; then
   if compile_prog "" "-lzbc" "libzbc"; then
-    minimum_libzbc=5
-    if $(pkg-config --atleast-version=$minimum_libzbc libzbc); then
-      libzbc="yes"
-    else
-      print_config "libzbc engine" "libzbc version $minimum_libzbc or above required"
+    libzbc="yes"
+    if ! check_min_lib_version libzbc 5; then
       libzbc="no"
     fi
   else
-- 
2.21.0

[PATCH v3 3/4] zbd: simplify zone reset code

[Dmitry Fomichev]

zbd_reset_range() function is only called once from zbd_reset_zone()
and it is always called for an individual zone, not a range.

Make zone reset flow simpler by moving all the code needed
to reset a single zone from zbd_reset_range() to zbd_reset_zone().
Therefore, zbd_reset_range() is now dropped.

zbc_reset_zone() is always called with the zone already locked. Remove
recursive zone locking inside this function and add a note in the
description of this function saying that the caller is expected to have
the zone locked when calling it.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
---
 zbd.c | 76 +++++++++++++++++++++--------------------------------------
 1 file changed, 27 insertions(+), 49 deletions(-)

diff --git a/zbd.c b/zbd.c
index 3eac5df3..e4a480b7 100644
--- a/zbd.c
+++ b/zbd.c
@@ -670,54 +670,6 @@ int zbd_setup_files(struct thread_data *td)
 	return 0;
 }
 
-/**
- * zbd_reset_range - reset zones for a range of sectors
- * @td: FIO thread data.
- * @f: Fio file for which to reset zones
- * @sector: Starting sector in units of 512 bytes
- * @nr_sectors: Number of sectors in units of 512 bytes
- *
- * Returns 0 upon success and a negative error code upon failure.
- */
-static int zbd_reset_range(struct thread_data *td, struct fio_file *f,
-			   uint64_t offset, uint64_t length)
-{
-	uint32_t zone_idx_b, zone_idx_e;
-	struct fio_zone_info *zb, *ze, *z;
-	int ret = 0;
-
-	assert(is_valid_offset(f, offset + length - 1));
-
-	switch (f->zbd_info->model) {
-	case ZBD_HOST_AWARE:
-	case ZBD_HOST_MANAGED:
-		ret = zbd_reset_wp(td, f, offset, length);
-		if (ret < 0)
-			return ret;
-		break;
-	default:
-		break;
-	}
-
-	zone_idx_b = zbd_zone_idx(f, offset);
-	zb = &f->zbd_info->zone_info[zone_idx_b];
-	zone_idx_e = zbd_zone_idx(f, offset + length);
-	ze = &f->zbd_info->zone_info[zone_idx_e];
-	for (z = zb; z < ze; z++) {
-		pthread_mutex_lock(&z->mutex);
-		pthread_mutex_lock(&f->zbd_info->mutex);
-		f->zbd_info->sectors_with_data -= z->wp - z->start;
-		pthread_mutex_unlock(&f->zbd_info->mutex);
-		z->wp = z->start;
-		z->verify_block = 0;
-		pthread_mutex_unlock(&z->mutex);
-	}
-
-	td->ts.nr_zone_resets += ze - zb;
-
-	return ret;
-}
-
 static unsigned int zbd_zone_nr(struct zoned_block_device_info *zbd_info,
 				struct fio_zone_info *zone)
 {
@@ -731,14 +683,40 @@ static unsigned int zbd_zone_nr(struct zoned_block_device_info *zbd_info,
  * @z: Zone to reset.
  *
  * Returns 0 upon success and a negative error code upon failure.
+ *
+ * The caller must hold z->mutex.
  */
 static int zbd_reset_zone(struct thread_data *td, struct fio_file *f,
 			  struct fio_zone_info *z)
 {
+	uint64_t offset = z->start;
+	uint64_t length = (z+1)->start - offset;
+	int ret = 0;
+
+	assert(is_valid_offset(f, offset + length - 1));
+
 	dprint(FD_ZBD, "%s: resetting wp of zone %u.\n", f->file_name,
 		zbd_zone_nr(f->zbd_info, z));
+	switch (f->zbd_info->model) {
+	case ZBD_HOST_AWARE:
+	case ZBD_HOST_MANAGED:
+		ret = zbd_reset_wp(td, f, offset, length);
+		if (ret < 0)
+			return ret;
+		break;
+	default:
+		break;
+	}
 
-	return zbd_reset_range(td, f, z->start, zbd_zone_end(z) - z->start);
+	pthread_mutex_lock(&f->zbd_info->mutex);
+	f->zbd_info->sectors_with_data -= z->wp - z->start;
+	pthread_mutex_unlock(&f->zbd_info->mutex);
+	z->wp = z->start;
+	z->verify_block = 0;
+
+	td->ts.nr_zone_resets++;
+
+	return ret;
 }
 
 /* The caller must hold f->zbd_info->mutex */
-- 
2.21.0

[PATCH v3 4/4] t/zbd: check log file for failed assertions

[Dmitry Fomichev]

Currently, a ZBD test can succeed even if an fio assertion is raised
during its run. Search every ZBD test log file for failed assertions
and fail the test if any were found.

Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
---
 t/zbd/test-zbd-support | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/t/zbd/test-zbd-support b/t/zbd/test-zbd-support
index 471a3487..139495d3 100755
--- a/t/zbd/test-zbd-support
+++ b/t/zbd/test-zbd-support
@@ -77,6 +77,13 @@ check_reset_count() {
     eval "[ '$reset_count' '$1' '$2' ]"
 }
 
+# Check log for failed assertions and crashes. Without these checks,
+# a test can succeed even when these events happen, but it must fail.
+check_log() {
+     [ ! -f "${logfile}.${1}" ] && return 0
+     ! grep -q -e "Assertion " -e "Aborted " "${logfile}.${1}"
+}
+
 # Whether or not $1 (/dev/...) is a SCSI device.
 is_scsi_device() {
     local d f
@@ -1008,7 +1015,7 @@ trap 'intr=1' SIGINT
 for test_number in "${tests[@]}"; do
     rm -f "${logfile}.${test_number}"
     echo -n "Running test $(printf "%02d" $test_number) ... "
-    if eval "test$test_number"; then
+    if eval "test$test_number" && check_log $test_number; then
 	status="PASS"
 	cc_status="${green}${status}${end}"
 	((passed++))
-- 
2.21.0

Re: [PATCH v3 1/4] configure: improve libzbc version check

[Damien Le Moal]

On 2020/08/04 10:38, Dmitry Fomichev wrote:
> Avoid parsing pkg-config output and just use --atleast-version to
> check if libzbc is present and has an up to date version.
> 
> Currently, support for libzbc ioengine is always included if libzbc is
> found at the build system. Add the option to disable libzbc ioengine
> support even if libzbc is found. This can be useful for
> cross-compilation.
> 
> Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
> ---
>  configure | 29 ++++++++++++++++-------------
>  1 file changed, 16 insertions(+), 13 deletions(-)
> 
> diff --git a/configure b/configure
> index 5925e94f..81fd32bb 100755
> --- a/configure
> +++ b/configure
> @@ -152,6 +152,7 @@ march_set="no"
>  libiscsi="no"
>  libnbd="no"
>  libaio_uring="no"
> +libzbc=""
>  dynamic_engines="no"
>  prefix=/usr/local
>  
> @@ -213,6 +214,8 @@ for opt do
>    ;;
>    --enable-libnbd) libnbd="yes"
>    ;;
> +  --disable-libzbc) libzbc="no"
> +  ;;
>    --disable-tcmalloc) disable_tcmalloc="yes"
>    ;;
>    --enable-libaio-uring) libaio_uring="yes"
> @@ -256,6 +259,7 @@ if test "$show_help" = "yes" ; then
>    echo "--with-ime=             Install path for DDN's Infinite Memory Engine"
>    echo "--enable-libiscsi       Enable iscsi support"
>    echo "--enable-libnbd         Enable libnbd (NBD engine) support"
> +  echo "--disable-libzbc        Disable libzbc even if found"
>    echo "--disable-tcmalloc	Disable tcmalloc support"
>    echo "--enable-libaio-uring   Enable libaio emulated over io_uring"
>    echo "--dynamic-libengines	Lib-based ioengines as dynamic libraries"
> @@ -2454,9 +2458,6 @@ fi
>  
>  ##########################################
>  # libzbc probe
> -if test "$libzbc" != "yes" ; then
> -  libzbc="no"
> -fi
>  cat > $TMPC << EOF
>  #include <libzbc/zbc.h>
>  int main(int argc, char **argv)
> @@ -2466,19 +2467,21 @@ int main(int argc, char **argv)
>    return zbc_open("foo=bar", O_RDONLY, &dev);
>  }
>  EOF
> -if compile_prog "" "-lzbc" "libzbc"; then
> -  libzbcvermaj=$(pkg-config --modversion libzbc | sed 's/\.[0-9]*\.[0-9]*//')
> -  if test "$libzbcvermaj" -ge "5" ; then
> -    libzbc="yes"
> +if test "$libzbc" != "no" ; then
> +  if compile_prog "" "-lzbc" "libzbc"; then
> +    minimum_libzbc=5
> +    if $(pkg-config --atleast-version=$minimum_libzbc libzbc); then
> +      libzbc="yes"
> +    else
> +      print_config "libzbc engine" "libzbc version $minimum_libzbc or above required"
> +      libzbc="no"
> +    fi
>    else
> -    print_config "libzbc engine" "Unsupported libzbc version (version 5 or above required)"
> +    if test "$libzbc" = "yes" ; then
> +      feature_not_found "libzbc" "libzbc or libzbc/zbc.h"
> +    fi
>      libzbc="no"
>    fi
> -else
> -  if test "$libzbc" = "yes" ; then
> -      feature_not_found "libzbc" "libzbc or libzbc/zbc.h"
> -  fi
> -  libzbc="no"
>  fi
>  print_config "libzbc engine" "$libzbc"
>  
> 

Looks good.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>

-- 
Damien Le Moal
Western Digital Research

Re: [PATCH v3 2/4] configure: check if pkg-config is installed

[Damien Le Moal]

On 2020/08/04 10:38, Dmitry Fomichev wrote:
> A few libraries need to be newer than a specific version in order to be
> supported by fio and pkg-config utility is used to verify the versions
> of the installed libraries. Since this step may fail because pkg-config
> is not installed, verify pkg-config presence and warn the user if it
> could not be found.
> 
> To avoid code duplication, add a common helper function to perform
> these checks.
> 
> Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
> ---
>  configure | 45 +++++++++++++++++++++++++--------------------
>  1 file changed, 25 insertions(+), 20 deletions(-)
> 
> diff --git a/configure b/configure
> index 81fd32bb..dd7fe3d2 100755
> --- a/configure
> +++ b/configure
> @@ -133,6 +133,23 @@ output_sym() {
>    echo "#define $1" >> $config_host_h
>  }
>  
> +check_min_lib_version() {
> +  local feature=$3
> +
> +  if ${cross_prefix}pkg-config --atleast-version=$2 $1 > /dev/null 2>&1; then
> +    return 0
> +  fi
> +  : ${feature:=${1}}
> +  if ${cross_prefix}pkg-config --version > /dev/null 2>&1; then
> +    if test ${!feature} = "yes" ; then
> +      feature_not_found "$feature" "$1 >= $2"
> +    fi
> +  else
> +    print_config "$1" "missing pkg-config, can't check $feature version"
> +  fi
> +  return 1
> +}
> +
>  targetos=""
>  cpu=""
>  
> @@ -1521,18 +1538,17 @@ if test "$?" != "0" ; then
>    echo "configure: gtk and gthread not found"
>    exit 1
>  fi
> -if ! ${cross_prefix}pkg-config --atleast-version 2.18.0 gtk+-2.0; then
> -  echo "GTK found, but need version 2.18 or higher"
> -  gfio="no"
> -else
> +gfio="yes"
> +if check_min_lib_version gtk+-2.0 2.18.0 "gfio"; then
>    if compile_prog "$GTK_CFLAGS" "$GTK_LIBS" "gfio" ; then
> -    gfio="yes"
>      GFIO_LIBS="$LIBS $GTK_LIBS"
>      CFLAGS="$CFLAGS $GTK_CFLAGS"
>    else
>      echo "Please install gtk and gdk libraries"
>      gfio="no"
>    fi
> +else
> +  gfio="no"
>  fi
>  LDFLAGS=$ORG_LDFLAGS
>  fi
> @@ -2182,15 +2198,11 @@ print_config "DDN's Infinite Memory Engine" "$libime"
>  ##########################################
>  # Check if we have libiscsi
>  if test "$libiscsi" != "no" ; then
> -  minimum_libiscsi=1.9.0
> -  if $(pkg-config --atleast-version=$minimum_libiscsi libiscsi); then
> +  if check_min_lib_version libiscsi 1.9.0; then
>      libiscsi="yes"
>      libiscsi_cflags=$(pkg-config --cflags libiscsi)
>      libiscsi_libs=$(pkg-config --libs libiscsi)
>    else
> -    if test "$libiscsi" = "yes" ; then
> -      feature_not_found "libiscsi" "libiscsi >= $minimum_libiscsi"
> -    fi
>      libiscsi="no"
>    fi
>  fi
> @@ -2199,15 +2211,11 @@ print_config "iscsi engine" "$libiscsi"
>  ##########################################
>  # Check if we have libnbd (for NBD support)
>  if test "$libnbd" != "no" ; then
> -  minimum_libnbd=0.9.8
> -  if $(pkg-config --atleast-version=$minimum_libnbd libnbd); then
> +  if check_min_lib_version libnbd 0.9.8; then
>      libnbd="yes"
>      libnbd_cflags=$(pkg-config --cflags libnbd)
>      libnbd_libs=$(pkg-config --libs libnbd)
>    else
> -    if test "$libnbd" = "yes" ; then
> -      feature_not_found "libnbd" "libnbd >= $minimum_libnbd"
> -    fi
>      libnbd="no"
>    fi
>  fi
> @@ -2469,11 +2477,8 @@ int main(int argc, char **argv)
>  EOF
>  if test "$libzbc" != "no" ; then
>    if compile_prog "" "-lzbc" "libzbc"; then
> -    minimum_libzbc=5
> -    if $(pkg-config --atleast-version=$minimum_libzbc libzbc); then
> -      libzbc="yes"
> -    else
> -      print_config "libzbc engine" "libzbc version $minimum_libzbc or above required"
> +    libzbc="yes"
> +    if ! check_min_lib_version libzbc 5; then
>        libzbc="no"
>      fi
>    else
> 

Looks good.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>

-- 
Damien Le Moal
Western Digital Research

RE: [PATCH v3 0/4] miscellaneous ZBD improvements

[Dmitry Fomichev]

Jens,

A gentle ping... Please consider this patchset for merging.

Very best,
Dmitry

> -----Original Message-----
> From: Dmitry Fomichev <dmitry.fomichev@wdc.com>
> Sent: Monday, August 3, 2020 9:38 PM
> To: Jens Axboe <axboe@kernel.dk>
> Cc: fio@vger.kernel.org; Damien Le Moal <Damien.LeMoal@wdc.com>;
> Sitsofe Wheeler <sitsofe@gmail.com>; Shinichiro Kawasaki
> <shinichiro.kawasaki@wdc.com>; Dmitry Fomichev
> <Dmitry.Fomichev@wdc.com>
> Subject: [PATCH v3 0/4] miscellaneous ZBD improvements
> 
> This series contains four patches related to zoned block devices.
> 
> v2 -> v3:
>   - initialized libzbc ioengine feature flag in configure
>   - improved error output in check_min_lib_version()
> 
> v1 -> v2:
>   - removed recursive zone locking in zbd_reset_zone()
>   - added --disable-libzbc configure option
>   - moved pkg-config version check to a separate function
>   - added a patch to verify that pkg-config is available to check
>     the minimum supported version in all libraries that need that
> 
> Dmitry Fomichev (4):
>   configure: improve libzbc version check
>   configure: check if pkg-config is installed
>   zbd: simplify zone reset code
>   t/zbd: check log file for failed assertions
> 
>  configure              | 62 +++++++++++++++++++---------------
>  t/zbd/test-zbd-support |  9 ++++-
>  zbd.c                  | 76 +++++++++++++++---------------------------
>  3 files changed, 70 insertions(+), 77 deletions(-)
> 
> --
> 2.21.0

Re: [PATCH v3 0/4] miscellaneous ZBD improvements

[Jens Axboe]

On 8/11/20 9:58 AM, Dmitry Fomichev wrote:
> Jens,
> 
> A gentle ping... Please consider this patchset for merging.

Applied, thanks.

BTW, can I get you to run your usual verification workloads with
zbd? I made some changes yesterday to fix a bug, would be nice
to ensure that zbd still works..
.
-- 
Jens Axboe

RE: [PATCH v3 0/4] miscellaneous ZBD improvements

[Dmitry Fomichev]

I ran ZBD tests against the zoned drives I have available and everything
looks good. A more extensive test suite will be run next Monday. I'll let
you know if something abnormal happens during that testing...

Dmitry

> -----Original Message-----
> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf
> Of Jens Axboe
> Sent: Tuesday, August 11, 2020 12:43 PM
> To: Dmitry Fomichev <Dmitry.Fomichev@wdc.com>
> Cc: fio@vger.kernel.org; Damien Le Moal <Damien.LeMoal@wdc.com>;
> Sitsofe Wheeler <sitsofe@gmail.com>; Shinichiro Kawasaki
> <shinichiro.kawasaki@wdc.com>
> Subject: Re: [PATCH v3 0/4] miscellaneous ZBD improvements
> 
> On 8/11/20 9:58 AM, Dmitry Fomichev wrote:
> > Jens,
> >
> > A gentle ping... Please consider this patchset for merging.
> 
> Applied, thanks.
> 
> BTW, can I get you to run your usual verification workloads with
> zbd? I made some changes yesterday to fix a bug, would be nice
> to ensure that zbd still works..
> .
> --
> Jens Axboe

Re: [PATCH v3 2/4] configure: check if pkg-config is installed

[Tomohiro Kusumi]

2020年8月5日(水) 16:06 Damien Le Moal <Damien.LeMoal@wdc.com>:
>
> On 2020/08/04 10:38, Dmitry Fomichev wrote:
> > A few libraries need to be newer than a specific version in order to be
> > supported by fio and pkg-config utility is used to verify the versions
> > of the installed libraries. Since this step may fail because pkg-config
> > is not installed, verify pkg-config presence and warn the user if it
> > could not be found.
> >
> > To avoid code duplication, add a common helper function to perform
> > these checks.

This line
"if test ${!feature} = "yes" ; then"
seems to have broken compilation on some BSD's.

[root@localhost fio]# uname
NetBSD
[root@localhost fio]# gmake clean && gmake
FIO_VERSION = fio-3.21-56-g0ce7e
Running configure ...
./configure: 144: Syntax error: Bad substitution
Makefile:19: config-host.mak: No such file or directory
gmake: *** [Makefile:10: config-host.mak] Error 2


> >
> > Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
> > ---
> >  configure | 45 +++++++++++++++++++++++++--------------------
> >  1 file changed, 25 insertions(+), 20 deletions(-)
> >
> > diff --git a/configure b/configure
> > index 81fd32bb..dd7fe3d2 100755
> > --- a/configure
> > +++ b/configure
> > @@ -133,6 +133,23 @@ output_sym() {
> >    echo "#define $1" >> $config_host_h
> >  }
> >
> > +check_min_lib_version() {
> > +  local feature=$3
> > +
> > +  if ${cross_prefix}pkg-config --atleast-version=$2 $1 > /dev/null 2>&1; then
> > +    return 0
> > +  fi
> > +  : ${feature:=${1}}
> > +  if ${cross_prefix}pkg-config --version > /dev/null 2>&1; then
> > +    if test ${!feature} = "yes" ; then
> > +      feature_not_found "$feature" "$1 >= $2"
> > +    fi
> > +  else
> > +    print_config "$1" "missing pkg-config, can't check $feature version"
> > +  fi
> > +  return 1
> > +}
> > +
> >  targetos=""
> >  cpu=""
> >
> > @@ -1521,18 +1538,17 @@ if test "$?" != "0" ; then
> >    echo "configure: gtk and gthread not found"
> >    exit 1
> >  fi
> > -if ! ${cross_prefix}pkg-config --atleast-version 2.18.0 gtk+-2.0; then
> > -  echo "GTK found, but need version 2.18 or higher"
> > -  gfio="no"
> > -else
> > +gfio="yes"
> > +if check_min_lib_version gtk+-2.0 2.18.0 "gfio"; then
> >    if compile_prog "$GTK_CFLAGS" "$GTK_LIBS" "gfio" ; then
> > -    gfio="yes"
> >      GFIO_LIBS="$LIBS $GTK_LIBS"
> >      CFLAGS="$CFLAGS $GTK_CFLAGS"
> >    else
> >      echo "Please install gtk and gdk libraries"
> >      gfio="no"
> >    fi
> > +else
> > +  gfio="no"
> >  fi
> >  LDFLAGS=$ORG_LDFLAGS
> >  fi
> > @@ -2182,15 +2198,11 @@ print_config "DDN's Infinite Memory Engine" "$libime"
> >  ##########################################
> >  # Check if we have libiscsi
> >  if test "$libiscsi" != "no" ; then
> > -  minimum_libiscsi=1.9.0
> > -  if $(pkg-config --atleast-version=$minimum_libiscsi libiscsi); then
> > +  if check_min_lib_version libiscsi 1.9.0; then
> >      libiscsi="yes"
> >      libiscsi_cflags=$(pkg-config --cflags libiscsi)
> >      libiscsi_libs=$(pkg-config --libs libiscsi)
> >    else
> > -    if test "$libiscsi" = "yes" ; then
> > -      feature_not_found "libiscsi" "libiscsi >= $minimum_libiscsi"
> > -    fi
> >      libiscsi="no"
> >    fi
> >  fi
> > @@ -2199,15 +2211,11 @@ print_config "iscsi engine" "$libiscsi"
> >  ##########################################
> >  # Check if we have libnbd (for NBD support)
> >  if test "$libnbd" != "no" ; then
> > -  minimum_libnbd=0.9.8
> > -  if $(pkg-config --atleast-version=$minimum_libnbd libnbd); then
> > +  if check_min_lib_version libnbd 0.9.8; then
> >      libnbd="yes"
> >      libnbd_cflags=$(pkg-config --cflags libnbd)
> >      libnbd_libs=$(pkg-config --libs libnbd)
> >    else
> > -    if test "$libnbd" = "yes" ; then
> > -      feature_not_found "libnbd" "libnbd >= $minimum_libnbd"
> > -    fi
> >      libnbd="no"
> >    fi
> >  fi
> > @@ -2469,11 +2477,8 @@ int main(int argc, char **argv)
> >  EOF
> >  if test "$libzbc" != "no" ; then
> >    if compile_prog "" "-lzbc" "libzbc"; then
> > -    minimum_libzbc=5
> > -    if $(pkg-config --atleast-version=$minimum_libzbc libzbc); then
> > -      libzbc="yes"
> > -    else
> > -      print_config "libzbc engine" "libzbc version $minimum_libzbc or above required"
> > +    libzbc="yes"
> > +    if ! check_min_lib_version libzbc 5; then
> >        libzbc="no"
> >      fi
> >    else
> >
>
> Looks good.
>
> Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
>
> --
> Damien Le Moal
> Western Digital Research

Re: [PATCH v3 2/4] configure: check if pkg-config is installed

[Sitsofe Wheeler]

On Sat, 15 Aug 2020 at 23:05, Tomohiro Kusumi <kusumi.tomohiro@gmail.com> wrote:
>
> 2020年8月5日(水) 16:06 Damien Le Moal <Damien.LeMoal@wdc.com>:
> >
> > On 2020/08/04 10:38, Dmitry Fomichev wrote:
> > > A few libraries need to be newer than a specific version in order to be
> > > supported by fio and pkg-config utility is used to verify the versions
> > > of the installed libraries. Since this step may fail because pkg-config
> > > is not installed, verify pkg-config presence and warn the user if it
> > > could not be found.
> > >
> > > To avoid code duplication, add a common helper function to perform
> > > these checks.
>
> This line
> "if test ${!feature} = "yes" ; then"
> seems to have broken compilation on some BSD's.
>
> [root@localhost fio]# uname
> NetBSD
> [root@localhost fio]# gmake clean && gmake
> FIO_VERSION = fio-3.21-56-g0ce7e
> Running configure ...
> ./configure: 144: Syntax error: Bad substitution
> Makefile:19: config-host.mak: No such file or directory
> gmake: *** [Makefile:10: config-host.mak] Error 2

Looks like a few bashisms have snuck into configure (shellcheck warns
about some other stuff). I think NetBSD's sh is ash...

-- 
Sitsofe | http://sucs.org/~sits/

RE: [PATCH v3 2/4] configure: check if pkg-config is installed

[Dmitry Fomichev]

> -----Original Message-----
> From: Sitsofe Wheeler <sitsofe@gmail.com>
> Sent: Sunday, August 16, 2020 4:48 PM
> To: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
> Cc: Damien Le Moal <Damien.LeMoal@wdc.com>; Dmitry Fomichev
> <Dmitry.Fomichev@wdc.com>; fio@vger.kernel.org
> Subject: Re: [PATCH v3 2/4] configure: check if pkg-config is installed
> 
> On Sat, 15 Aug 2020 at 23:05, Tomohiro Kusumi
> <kusumi.tomohiro@gmail.com> wrote:
> >
> > 2020年8月5日(水) 16:06 Damien Le Moal <Damien.LeMoal@wdc.com>:
> > >
> > > On 2020/08/04 10:38, Dmitry Fomichev wrote:
> > > > A few libraries need to be newer than a specific version in order to be
> > > > supported by fio and pkg-config utility is used to verify the versions
> > > > of the installed libraries. Since this step may fail because pkg-config
> > > > is not installed, verify pkg-config presence and warn the user if it
> > > > could not be found.
> > > >
> > > > To avoid code duplication, add a common helper function to perform
> > > > these checks.
> >
> > This line
> > "if test ${!feature} = "yes" ; then"
> > seems to have broken compilation on some BSD's.
> >
> > [root@localhost fio]# uname
> > NetBSD
> > [root@localhost fio]# gmake clean && gmake
> > FIO_VERSION = fio-3.21-56-g0ce7e
> > Running configure ...
> > ./configure: 144: Syntax error: Bad substitution
> > Makefile:19: config-host.mak: No such file or directory
> > gmake: *** [Makefile:10: config-host.mak] Error 2
> 
> Looks like a few bashisms have snuck into configure (shellcheck warns
> about some other stuff). I think NetBSD's sh is ash...
> 

Indeed. I just sent the fix for this problem.

Tomohiro,
Would you be able to test the new patch?

Thank you,
Dmitry


> --
> Sitsofe | http://sucs.org/~sits/

Re: [PATCH v3 2/4] configure: check if pkg-config is installed

[Tomohiro Kusumi]

2020年8月18日(火) 8:50 Dmitry Fomichev <Dmitry.Fomichev@wdc.com>:
>
>
>
> > -----Original Message-----
> > From: Sitsofe Wheeler <sitsofe@gmail.com>
> > Sent: Sunday, August 16, 2020 4:48 PM
> > To: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
> > Cc: Damien Le Moal <Damien.LeMoal@wdc.com>; Dmitry Fomichev
> > <Dmitry.Fomichev@wdc.com>; fio@vger.kernel.org
> > Subject: Re: [PATCH v3 2/4] configure: check if pkg-config is installed
> >
> > On Sat, 15 Aug 2020 at 23:05, Tomohiro Kusumi
> > <kusumi.tomohiro@gmail.com> wrote:
> > >
> > > 2020年8月5日(水) 16:06 Damien Le Moal <Damien.LeMoal@wdc.com>:
> > > >
> > > > On 2020/08/04 10:38, Dmitry Fomichev wrote:
> > > > > A few libraries need to be newer than a specific version in order to be
> > > > > supported by fio and pkg-config utility is used to verify the versions
> > > > > of the installed libraries. Since this step may fail because pkg-config
> > > > > is not installed, verify pkg-config presence and warn the user if it
> > > > > could not be found.
> > > > >
> > > > > To avoid code duplication, add a common helper function to perform
> > > > > these checks.
> > >
> > > This line
> > > "if test ${!feature} = "yes" ; then"
> > > seems to have broken compilation on some BSD's.
> > >
> > > [root@localhost fio]# uname
> > > NetBSD
> > > [root@localhost fio]# gmake clean && gmake
> > > FIO_VERSION = fio-3.21-56-g0ce7e
> > > Running configure ...
> > > ./configure: 144: Syntax error: Bad substitution
> > > Makefile:19: config-host.mak: No such file or directory
> > > gmake: *** [Makefile:10: config-host.mak] Error 2
> >
> > Looks like a few bashisms have snuck into configure (shellcheck warns
> > about some other stuff). I think NetBSD's sh is ash...
> >
>
> Indeed. I just sent the fix for this problem.
>
> Tomohiro,
> Would you be able to test the new patch?
>
> Thank you,
> Dmitry

Thanks, I've confirmed ./configure issue is fixed on NetBSD.