[PATCH 0/6] RFC add blkdev tests v2

[PATCH 0/6] RFC add blkdev tests v2

[Dmitry Monakhov]

During LSFMM we have discussed how to test lower-backend of linux IO-stack.
Main idea was that xfstests is the most obvious solution which cover
most use-cases filesystem do care about. There were reasonable objections
that xfstests is not ideal place for that, especially when they mixed
with pure FS specific ones.

My opinion is that it is worth to have blkdev tests here, in xfstests,
instead of not have it at all. But tests have to be placed to separate
namespace "tests/blockdev" to prevent confusion.

Please review my RFC and tell me: Is my proposal too ugly? And we have to
have completely separate framework for basic blockdev sanity checks.
Assumptions:
  - All new feature and regression tests should goes to "tests/blockdev"
  - All tests belongs to 'blockdev' group
  - Pure blkdev tests (tests which behavior is independent from fs) is not
    added to 'auto' group. So it will not affect normal fs testing flow.

TOC:
0001-add-lio-target-helpers.patch
0002-add-blockdev-001-check-page-cache-coherency-after-BLKDISCARD
0003-new-blockdev-002-check-information-leak-for-lio-file
0004-new-blockdev-003-basic-blockdev-T10-DIF-TYPE1-integr
0005-new-blockdev-004-Check-that-live-fs-survives-blkdev
0006-new-blockdev-005-Check-that-busy-fs-survives-blkdev

[PATCH 1/6] add lio-target helpers

[Dmitry Monakhov]

Linux-IO Target is very good framework for testing block backend.
It is more flexible than scsi_debug.

http://linux-iscsi.org/wiki/LIO
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 common/config    |   2 +
 common/liotarget | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 113 insertions(+)
 create mode 100644 common/liotarget

diff --git a/common/config b/common/config
index 59041a3..cfe7913 100644
--- a/common/config
+++ b/common/config
@@ -212,6 +212,8 @@ export XZ_PROG="`set_prog_path xz`"
 export FLOCK_PROG="`set_prog_path flock`"
 export LDD_PROG="`set_prog_path ldd`"
 export TIMEOUT_PROG="`set_prog_path timeout`"
+export TARGETCLI_PROG="`set_prog_path targetcli`"
+export TARGETCTL_PROG="`set_prog_path targetctl`"
 
 # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
 # newer systems have udevadm command but older systems like RHEL5 don't.
diff --git a/common/liotarget b/common/liotarget
new file mode 100644
index 0000000..f821692
--- /dev/null
+++ b/common/liotarget
@@ -0,0 +1,111 @@
+#!/bin/bash
+#
+# Copyright (c) 2017 Virtuozzo Inc
+# All Rights Reserved.
+#
+# Written by Dmitry Monakhov <dmonakhov@openvz.org>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#
+# Functions for Linux-IO Target manipulation
+#
+
+_require_liotarget()
+{
+	which $TARGETCLI_PROG >>$seqres.full 2>&1  || \
+	    _notrun "this test requires 'targetcli' tool"
+	which $TARGETCLI_PROG >>$seqres.full 2>&1  || \
+	    _notrun "this test requires 'targetcli' tool"
+
+	$TARGETCLI_PROG ls /backstores/ramdisk >>$seqres.full 2>&1 ||\
+	    _notrun "kernel compiled w/o CONFIG_TARGET_CORE"
+	$TARGETCLI_PROG ls /backstores/fileio >>$seqres.full 2>&1 ||\
+	    _notrun "kernel compiled w/o CONFIG_TCM_FILEIO"
+	$TARGETCLI_PROG ls /loopback >>$seqres.full 2>&1 ||\
+	    _notrun "kernel compiled w/o CONFIG_LOOPBACK_TARGET"
+}
+
+_liotgt_create_fileio()
+{
+	local name=$1
+	local img=$2
+	local sz=$3
+
+	$TARGETCLI_PROG /backstores/fileio create \
+			name=$name file_or_dev=$img  size=$sz >>$seqres.full ||\
+	    _fail "Can not create /backstores/fileio/$name"
+
+	local cfg_path=`ls -d /sys/kernel/config/target/core/fileio_*/$name`
+	[ -d "$cfg_path" ] || _fail "Bad config path"
+	echo $cfg_path
+}
+
+_liotgt_set_attribute()
+{
+	local path=$1
+	local attr_name=$2
+	local attr_val=$3
+
+	[ -f $path/attrib/$attr_name ] || _fail "Bad attribute $attr_name"
+	echo "echo $attr_val >  $path/attrib/$attr_name " >>$seqres.full
+	echo $attr_val >  $path/attrib/$attr_name
+}
+
+_liotgt_create_loopback()
+{
+	local out=`$TARGETCLI_PROG /loopback/ create 2>&1`
+	[ $? -eq 0 ] || _fail "Can not create loopback target"
+	echo $out >>$seqres.full
+
+	local naa=`echo $out | gawk '{ print $3 }'`
+	echo ${naa:0:20} >>$seqres.full
+
+	echo ${naa:0:20}
+}
+
+_liotgt_find_dev()
+{
+	local found=""
+	local name=$1
+	local drives=`find /sys/devices -type f -name model`
+	for d in $drives;do
+		local dir=`dirname $d`
+		local vendor=`cat $dir/vendor`
+		local model=`cat $dir/model`
+		if [ "${vendor//[[:space:]]/}" == "LIO-ORG" ] && \
+		       [ "${model//[[:space:]]/}" == "$name" ]; then
+			found=/dev/$(ls $dir/block)
+			break
+		fi
+	done
+	[ -z "$found" ] && _fail "Can not find device with backend $name"
+	echo "$found"
+}
+
+_liotgt_attach_target()
+{
+	local path=$1
+	local name=`basename $path`
+	local naa=$(_liotgt_create_loopback)
+
+	$TARGETCLI_PROG /loopback/$naa/luns create $path >>$seqres.full 2>&1 ||\
+	    _fail "Can not attach $path to tcm_loop"
+	_liotgt_find_dev $name
+}
+
+_liotgt_cleanup()
+{
+	$TARGETCTL_PROG clear
+}
-- 
2.9.3

[PATCH 2/6] add: blockdev/001 check page-cache coherency after BLKDISCARD

[Dmitry Monakhov]

Regression test for fix https://lkml.org/lkml/2017/3/22/789
Page cache should be dropped after successful discard

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 common/config          |  1 +
 tests/blockdev/001     | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/blockdev/001.out |  7 +++++
 tests/blockdev/group   |  6 ++++
 4 files changed, 88 insertions(+)
 create mode 100755 tests/blockdev/001
 create mode 100644 tests/blockdev/001.out
 create mode 100644 tests/blockdev/group

diff --git a/common/config b/common/config
index cfe7913..aef842c 100644
--- a/common/config
+++ b/common/config
@@ -199,6 +199,7 @@ export LOGGER_PROG="`set_prog_path logger`"
 export DBENCH_PROG="`set_prog_path dbench`"
 export DMSETUP_PROG="`set_prog_path dmsetup`"
 export WIPEFS_PROG="`set_prog_path wipefs`"
+export BLKDISCARD_PROG="`set_prog_path blkdiscard`"
 export DUMP_PROG="`set_prog_path dump`"
 export RESTORE_PROG="`set_prog_path restore`"
 export LVM_PROG="`set_prog_path lvm`"
diff --git a/tests/blockdev/001 b/tests/blockdev/001
new file mode 100755
index 0000000..53b0664
--- /dev/null
+++ b/tests/blockdev/001
@@ -0,0 +1,74 @@
+#! /bin/bash
+# FS QA Test 001
+#
+# Regression test for fix https://lkml.org/lkml/2017/3/22/789
+# Page cache should be dropped after successful discard
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Dmitry Monakhov <dmonakhov@openvz.org>
+# All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/scsi_debug
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scsi_debug
+_require_command $BLKDISCARD_PROG blkdiscard
+
+
+# Create virtual device with unmap_zeroes_data support
+dev=$(_get_scsi_debug_dev 512 512 0 32 "lbpws=1 lbpws10=1")
+
+# Initialize data with known pattern
+$XFS_IO_PROG -c "pwrite -S 0xaa -b 2M 0 32M -w" -d $dev >>$seqres.full 2>&1 || \
+    _fail "pwrite failed"
+
+# Fill page cache with dirty data
+$XFS_IO_PROG -c "pwrite -S 0xbb -b 2M 2M 10M" -f $dev >>$seqres.full 2>&1 || \
+    _fail "pwrite failed"
+
+$BLKDISCARD_PROG $dev
+# After device was fully discarded, read sould return all zeroes
+dd if=$dev bs=4k  2>>$seqres.full | hexdump
+dd if=$dev bs=4M iflag=direct 2>>$seqres.full | hexdump
+# success, all done
+status=0
+exit
diff --git a/tests/blockdev/001.out b/tests/blockdev/001.out
new file mode 100644
index 0000000..0f81a92
--- /dev/null
+++ b/tests/blockdev/001.out
@@ -0,0 +1,7 @@
+QA output created by 001
+0000000 0000 0000 0000 0000 0000 0000 0000 0000
+*
+2000000
+0000000 0000 0000 0000 0000 0000 0000 0000 0000
+*
+2000000
diff --git a/tests/blockdev/group b/tests/blockdev/group
new file mode 100644
index 0000000..bdfdcb9
--- /dev/null
+++ b/tests/blockdev/group
@@ -0,0 +1,6 @@
+# QA groups control file
+# Defines test groups and nominal group owners
+# - do not start group names with a digit
+# - comment line before each group is "new" description
+#
+001 rw blockdev
-- 
2.9.3

[PATCH 3/6] new: blockdev/002 check information leak for lio-fileio

[Dmitry Monakhov]

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 tests/blockdev/002     | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/blockdev/002.out | 12 ++++++++
 tests/blockdev/group   |  1 +
 3 files changed, 90 insertions(+)
 create mode 100755 tests/blockdev/002
 create mode 100644 tests/blockdev/002.out

diff --git a/tests/blockdev/002 b/tests/blockdev/002
new file mode 100755
index 0000000..71867d1
--- /dev/null
+++ b/tests/blockdev/002
@@ -0,0 +1,77 @@
+#! /bin/bash
+# FS QA Test 002
+#
+# Regression test for  information leak for lio-fileio target
+# BUG: If image  file is less than virtual blockdev then read() may return
+#      unitilized data.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Dmitry Monakhov <dmonakhov@openvz.org>
+# All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	_liotgt_cleanup
+	rm -rf $TEST_DIR/$$
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/liotarget
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_liotarget
+
+mkdir -p $TEST_DIR/$$ || _fail "Can not make test dir"
+
+cfg_path=$(_liotgt_create_fileio  002-test.img $TEST_DIR/$$/002-test.img 128M)
+dev=$(_liotgt_attach_target /backstores/fileio/002-test.img)
+
+$XFS_IO_PROG -f -c "truncate 1M" $TEST_DIR/$$/002-test.img >> $seqres.full
+
+$XFS_IO_PROG -c "pwrite -S 0xa0 -b 512 512 1k" -d $dev >>$seqres.full 2>&1 || \
+    _fail "pwrite failed"
+
+$XFS_IO_PROG -c "pwrite -S 0xab -b 512 2M 1k" -d $dev >>$seqres.full 2>&1 || \
+    _fail "pwrite failed"
+
+dd if=$dev bs=4k  2>>$seqres.full | hexdump
+# success, all done
+status=0
+exit
diff --git a/tests/blockdev/002.out b/tests/blockdev/002.out
new file mode 100644
index 0000000..e761c06
--- /dev/null
+++ b/tests/blockdev/002.out
@@ -0,0 +1,12 @@
+QA output created by 002
+0000000 0000 0000 0000 0000 0000 0000 0000 0000
+*
+0000200 a0a0 a0a0 a0a0 a0a0 a0a0 a0a0 a0a0 a0a0
+*
+0000600 0000 0000 0000 0000 0000 0000 0000 0000
+*
+0200000 abab abab abab abab abab abab abab abab
+*
+0200400 0000 0000 0000 0000 0000 0000 0000 0000
+*
+8000000
diff --git a/tests/blockdev/group b/tests/blockdev/group
index bdfdcb9..0b511f3 100644
--- a/tests/blockdev/group
+++ b/tests/blockdev/group
@@ -4,3 +4,4 @@
 # - comment line before each group is "new" description
 #
 001 rw blockdev
+002 rw blockdev liotarget
-- 
2.9.3

[PATCH 4/6] new: blockdev/003 basic blockdev T10-DIF-TYPE1 integrity checks

[Dmitry Monakhov]

Test create virtual block device via lio-targed infastructure and
perform basic IO operations with data corruption detection.

Temprorally mark is as dangerous, because currently it trigger BUG_ON
inside blkdev_issue_flush

BTW: I use 'dd' to test read from corrupted image instead of xfs_io
because even if pread failed, xfs_io still exit with success, BUG?

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 tests/blockdev/003     | 164 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/blockdev/003.out |  39 ++++++++++++
 tests/blockdev/group   |   1 +
 3 files changed, 204 insertions(+)
 create mode 100755 tests/blockdev/003
 create mode 100644 tests/blockdev/003.out

diff --git a/tests/blockdev/003 b/tests/blockdev/003
new file mode 100755
index 0000000..8f31f0d
--- /dev/null
+++ b/tests/blockdev/003
@@ -0,0 +1,164 @@
+#! /bin/bash
+# FS QA Test 003
+#
+# Check basic T10-DIF integrity features for a block device
+#
+# DIF/DIX TYPE: T10-DIF-TYPE1-CRC
+# Kernel docs: Documentation/blockdev/data-integrity.txt
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Dmitry Monakhov <dmonakhov@openvz.org>
+# All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	_liotgt_cleanup
+	rm -rf $TEST_DIR/$$
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/liotarget
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_liotarget
+
+mkdir -p $TEST_DIR/$$ || _fail "Can not make test dir"
+
+# One full test suite
+_run_one()
+{
+	local img_name=$1
+
+	echo "Run: $img_name"
+	echo "T0: Test basic IO"
+	$XFS_IO_PROG -c "pwrite -S 0xa0 -b 4M 0 16M" -d $dev >>$seqres.full 2>&1 || \
+	    _fail "pwrite failed"
+
+	$XFS_IO_PROG -c "pwrite -S 0xa3 -b 1k -V8  20M 32k" -d $dev >>$seqres.full 2>&1 || \
+	    _fail "pwrite failed"
+
+	$XFS_IO_PROG -c "pwrite -S 0xa2 -b 1k -V8  2M 32k" -f $dev >>$seqres.full 2>&1 || \
+	    _fail "pwrite failed"
+
+	$XFS_IO_PROG -c "pwrite -S 0xa1 -b 4k  1536000 8k" -f $dev >>$seqres.full 2>&1 || \
+	    _fail "pwrite failed"
+
+	$XFS_IO_PROG -c "fsync" -d $dev >>$seqres.full 2>&1 || _fail "fsync failed"
+
+	echo "Check that buffered IO"
+	dd if=$dev bs=4k  2>>$seqres.full | md5sum
+	echo "Check that direct IO"
+	# Small requests will not being splitted
+	dd if=$dev bs=64k iflag=direct 2>>$seqres.full | md5sum
+	# Big requests may be splitted on flight, but result MUST be the same.
+	dd if=$dev bs=4M iflag=direct 2>>$seqres.full | md5sum
+
+	echo "Check csum corruption detection"
+	# LIO-fileio store t10 DIF data in separate file ${IMG}.protection
+	# struct t10_pi_tuple {
+	#	__be16 guard_tag;       /* Checksum */
+	#       __be16 app_tag;         /* Opaque storage */
+	#       __be32 ref_tag;         /* Target LBA or indirect LBA */
+	#}
+	# Play with 3000'th sector -> t10_pi_tuple offset == 3000 * 8 == 24000
+	#
+	echo "T1: Corrupt guard_tag, next read should fail"
+	$XFS_IO_PROG -c "pwrite -S 0xde -b2 24000 2 -w" \
+		     -f $TEST_DIR/$$/$img_name.protection >>$seqres.full 2>&1
+	dd if=$dev of=/dev/null bs=1M count=2 iflag=direct >>$seqres.full 2>&1 &&
+	    _fail "read should fail on 3000'th sector"
+
+	echo "T2: Check that unaffected blocks are still readable"
+	dd if=$dev of=/dev/null bs=1M count=1 iflag=direct >>$seqres.full 2>&1 || _fail "read failed"
+
+	echo "T3: Rewrite corrupted sector and check that read works"
+	$XFS_IO_PROG -c "pwrite -S 0xa1 -b 4k 1536000 4k" -d $dev >>$seqres.full 2>&1 || \
+	    _fail "pwrite failed"
+	dd if=$dev bs=2M count=1 iflag=direct >>$seqres.full 2>&1 || _fail "read failed"
+
+	echo "T4: Corrupt app_tag, should not affect read"
+	$XFS_IO_PROG -c "pwrite -S 0xde -b2 24002 2 -w" \
+		     -f $TEST_DIR/$$/$img_name.protection >>$seqres.full 2>&1
+	dd if=$dev bs=2M count=1 iflag=direct >>$seqres.full 2>&1 || _fail "read failed"
+
+	echo "T5: Corrupt ref_tag, next read should fail"
+	$XFS_IO_PROG -c "pwrite -S 0xde -b4 24004 4 -w"  \
+		     $TEST_DIR/$$/$img_name.protection  >>$seqres.full 2>&1
+	dd if=$dev of=/dev/null bs=1M count=2 iflag=direct >>$seqres.full 2>&1 &&
+	    _fail "read should fail on 3000'th sector"
+
+	echo "T6: Check that unaffected blocks are still readable"
+	dd if=$dev of=/dev/null bs=1M count=1 iflag=direct >>$seqres.full 2>&1 || _fail "read failed"
+
+	echo "T7: Rewrite corrupted sector and check that read works"
+	$XFS_IO_PROG -c "pwrite -S 0xa1 -b 4k 1536000 4k" -d $dev >>$seqres.full 2>&1 || \
+	    _fail "pwrite failed"
+
+	echo "Check that buffered and direct read works"
+	dd if=$dev bs=4k  2>>$seqres.full | md5sum
+	# Small requests will not being splitted
+	dd if=$dev bs=64k iflag=direct 2>>$seqres.full | md5sum
+	# Big requests may be splitted on flight, but result MUST be the same.
+	dd if=$dev bs=4M iflag=direct 2>>$seqres.full | md5sum
+}
+
+# Create virtual scsi target with internal csum verification
+name=dif-type1-w
+cfg_path=$(_liotgt_create_fileio $name $TEST_DIR/$$/$name 32M)
+
+_liotgt_set_attribute $cfg_path  pi_prot_type 1
+_liotgt_set_attribute $cfg_path  pi_prot_format 1
+_liotgt_set_attribute $cfg_path  pi_prot_verify 1
+dev=$(_liotgt_attach_target /backstores/fileio/$name)
+_run_one $name
+
+# Create virtual scsi target w/o internal csum verification,
+# check that core blk_integrity_profile->verify_fn works
+#
+name=dif-type1-wo
+cfg_path=$(_liotgt_create_fileio $name $TEST_DIR/$$/$name 32M)
+
+_liotgt_set_attribute $cfg_path  pi_prot_type 1
+_liotgt_set_attribute $cfg_path  pi_prot_format 1
+_liotgt_set_attribute $cfg_path  pi_prot_verify 0
+dev=$(_liotgt_attach_target /backstores/fileio/$name)
+_run_one $name
+
+status=0
+exit
diff --git a/tests/blockdev/003.out b/tests/blockdev/003.out
new file mode 100644
index 0000000..be2efbb
--- /dev/null
+++ b/tests/blockdev/003.out
@@ -0,0 +1,39 @@
+QA output created by 003
+Run: dif-type1-w
+T0: Test basic IO
+Check that buffered IO
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+Check that direct IO
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+Check csum corruption detection
+T1: Corrupt guard_tag, next read should fail
+T2: Check that unaffected blocks are still readable
+T3: Rewrite corrupted sector and check that read works
+T4: Corrupt app_tag, should not affect read
+T5: Corrupt ref_tag, next read should fail
+T6: Check that unaffected blocks are still readable
+T7: Rewrite corrupted sector and check that read works
+Check that buffered and direct read works
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+Run: dif-type1-wo
+T0: Test basic IO
+Check that buffered IO
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+Check that direct IO
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+Check csum corruption detection
+T1: Corrupt guard_tag, next read should fail
+T2: Check that unaffected blocks are still readable
+T3: Rewrite corrupted sector and check that read works
+T4: Corrupt app_tag, should not affect read
+T5: Corrupt ref_tag, next read should fail
+T6: Check that unaffected blocks are still readable
+T7: Rewrite corrupted sector and check that read works
+Check that buffered and direct read works
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+f18a6c9b9d69ff8ed61daa62dfab4341  -
+f18a6c9b9d69ff8ed61daa62dfab4341  -
diff --git a/tests/blockdev/group b/tests/blockdev/group
index 0b511f3..e5ce864 100644
--- a/tests/blockdev/group
+++ b/tests/blockdev/group
@@ -5,3 +5,4 @@
 #
 001 rw blockdev
 002 rw blockdev liotarget
+003 rw blockdev liotarget integrity dangerous
-- 
2.9.3

[PATCH 5/6] new: blockdev/004 Check that live fs survives blkdev page truncation

[Dmitry Monakhov]

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 tests/blockdev/004     | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/blockdev/004.out |  2 ++
 tests/blockdev/group   |  1 +
 3 files changed, 84 insertions(+)
 create mode 100755 tests/blockdev/004
 create mode 100644 tests/blockdev/004.out

diff --git a/tests/blockdev/004 b/tests/blockdev/004
new file mode 100755
index 0000000..decb4ad
--- /dev/null
+++ b/tests/blockdev/004
@@ -0,0 +1,81 @@
+#! /bin/bash
+# FS QA Test 004
+#
+# Check that live filesystem survives blockdev pagecache truncation.
+# blkdev page cache can be truncated due to various reasons
+# 1)ioctl: BLKDISCARD
+# 2)blkdev: falloc{ FALLOC_FL_ZERO_RANGE, FALLOC_FL_PUNCH_HOLE }
+# 3)nbd: NBD_CLEAR_SOCK
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Dmitry Monakhov <dmonakhov@openvz.org>
+# All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	$UMOUNT_PROG $fs_mnt
+	_destroy_loop_device $loop_dev
+	cd /
+	rm -rf $TEST_DIR/$$
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_loop
+
+# create the fs image and mount point
+
+mkdir -p $TEST_DIR/$$ || _fail "Can not make test dir"
+fs_img=$TEST_DIR/$$/fs.img
+fs_mnt=$TEST_DIR/$$/mnt
+$XFS_IO_PROG -fc "truncate 1g" $fs_img >>$seqres.full 2>&1
+mkdir -p $fs_mnt
+loop_dev=$(_create_loop_device $fs_img)
+_mkfs_dev $loop_dev
+_mount $loop_dev $fs_mnt
+
+# Issue zeroout on block device, this act just a giant hammer for fs,
+# A lot of complains are expected, bug_on/panic is not.
+$XFS_IO_PROG -c "fzero -k 0 1G" -f $loop_dev >>$seqres.full 2>&1
+# Try IO on corrupted fs
+for ((i=0;i<1000;i++));do
+	$XFS_IO_PROG -fc "pwrite -S $i 0 16k" $fs_mnt/test-$i >>$seqres.full 2>&1
+done
+$XFS_IO_PROG -fc "fsync" $fs_mnt/test-0 >>$seqres.full 2>&1
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/blockdev/004.out b/tests/blockdev/004.out
new file mode 100644
index 0000000..af8614a
--- /dev/null
+++ b/tests/blockdev/004.out
@@ -0,0 +1,2 @@
+QA output created by 004
+Silence is golden
diff --git a/tests/blockdev/group b/tests/blockdev/group
index e5ce864..d100c85 100644
--- a/tests/blockdev/group
+++ b/tests/blockdev/group
@@ -6,3 +6,4 @@
 001 rw blockdev
 002 rw blockdev liotarget
 003 rw blockdev liotarget integrity dangerous
+004 rw blockdev dangerous zero
-- 
2.9.3

[PATCH 6/6] new: blockdev/005 Check that busy fs survives blkdev page truncation

[Dmitry Monakhov]

same as blockdev/004 but with add more stress on corrupted fs.

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 tests/blockdev/005     | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/blockdev/005.out |  2 ++
 tests/blockdev/group   |  1 +
 tests/generic/390      |  1 +
 4 files changed, 96 insertions(+)
 create mode 100755 tests/blockdev/005
 create mode 100644 tests/blockdev/005.out

diff --git a/tests/blockdev/005 b/tests/blockdev/005
new file mode 100755
index 0000000..fa3eadc
--- /dev/null
+++ b/tests/blockdev/005
@@ -0,0 +1,92 @@
+#! /bin/bash
+# FS QA Test 005
+#
+# Check that busy filesystem survives blockdev pagecache truncation.
+# blkdev page cache can be truncated due to various reasons
+# 1)ioctl: BLKDISCARD
+# 2)blkdev: falloc{ FALLOC_FL_ZERO_RANGE, FALLOC_FL_PUNCH_HOLE }
+# 3)nbd: NBD_CLEAR_SOCK
+# test is similar blockdev/004 but push more stress on filesystem.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Dmitry Monakhov <dmonakhov@openvz.org>
+# All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	$UMOUNT_PROG $fs_mnt
+	_destroy_loop_device $loop_dev
+	cd /
+	rm -rf $TEST_DIR/$$
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_loop
+#_require_xfs_io_command "fzero"
+
+# create the fs image and mount point
+
+mkdir -p $TEST_DIR/$$ || _fail "Can not make test dir"
+fs_img=$TEST_DIR/$$/fs.img
+fs_mnt=$TEST_DIR/$$/mnt
+$XFS_IO_PROG -fc "truncate 1g" $fs_img >>$seqres.full 2>&1
+mkdir -p $fs_mnt
+loop_dev=$(_create_loop_device $fs_img)
+_mkfs_dev $loop_dev
+_mount $loop_dev $fs_mnt
+
+stress_dir="$fs_mnt/stress"
+mkdir "$stress_dir"
+# Disable all sync operations to get higher load
+FSSTRESS_AVOID="$FSSTRESS_AVOID -ffsync=0 -fsync=0 -ffdatasync=0"
+fsstress_args=`_scale_fsstress_args -d $stress_dir -p4 -n9999999 $FSSTRESS_AVOID`
+$FSSTRESS_PROG $fsstress_args > /dev/null 2>&1 &
+fsstress_pid=$!
+# Warm-up
+sleep 3
+
+# Issue zeroout on block device, this act just a giant hammer for fs,
+# A lot of complains are expected, bug_on/panic is not.
+$XFS_IO_PROG -c "fzero -k 0 1G" -f $loop_dev >>$seqres.full 2>&1
+# Force sync operation in order to flush data to screwed disk.
+$XFS_IO_PROG -c "pwrite 0 4k" -c "syncfs" -f $fs_mnt/check_ro >>$seqres.full 2>&1
+sleep 3
+$KILLALL_PROG -q $FSSTRESS_PROG
+wait $pid
+# success, all done
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/blockdev/005.out b/tests/blockdev/005.out
new file mode 100644
index 0000000..a5027f1
--- /dev/null
+++ b/tests/blockdev/005.out
@@ -0,0 +1,2 @@
+QA output created by 005
+Silence is golden
diff --git a/tests/blockdev/group b/tests/blockdev/group
index d100c85..445b97c 100644
--- a/tests/blockdev/group
+++ b/tests/blockdev/group
@@ -7,3 +7,4 @@
 002 rw blockdev liotarget
 003 rw blockdev liotarget integrity dangerous
 004 rw blockdev dangerous zero
+005 rw blockdev dangerous zero
diff --git a/tests/generic/390 b/tests/generic/390
index 473015a..a717888 100755
--- a/tests/generic/390
+++ b/tests/generic/390
@@ -55,6 +55,7 @@ _supported_os Linux
 _require_scratch
 _require_freeze
 _require_test_program "feature"
+_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs >>$seqres.full 2>&1
 # We don't want to freeze/unfreeze root fs if mount scratch dev failed
-- 
2.9.3

Re: [PATCH 0/6] RFC add blkdev tests v2

[Christoph Hellwig]

I still disagree with using xfstests for this.  Even if we were going
to treat the block devices nodes as yet another file system that's not
what the patches do - they create specific virtual devices to test
for each test.

I think the right way is to keep your patches as-is and copy the few
bits you use from xfstests into a new repository.

Re: [PATCH 0/6] RFC add blkdev tests v2

[Jens Axboe]

On 04/06/2017 07:55 AM, Christoph Hellwig wrote:
> I still disagree with using xfstests for this.  Even if we were going
> to treat the block devices nodes as yet another file system that's not
> what the patches do - they create specific virtual devices to test
> for each test.
> 
> I think the right way is to keep your patches as-is and copy the few
> bits you use from xfstests into a new repository.

That is exactly what my recommendation was at lsfmm as well - fork
xfstest, prune bits we don't need, and off we go. I'll get around to
it soonish.

-- 
Jens Axboe

Re: [PATCH 0/6] RFC add blkdev tests v2

[Christoph Hellwig]

On Thu, Apr 06, 2017 at 08:33:55AM -0600, Jens Axboe wrote:
> That is exactly what my recommendation was at lsfmm as well - fork
> xfstest, prune bits we don't need, and off we go. I'll get around to
> it soonish.

So you volunteer to do it?  I was -><- this close to offering it myself,
and than thought of all the other backlog I have on my plate :)

Re: [PATCH 0/6] RFC add blkdev tests v2

[Jens Axboe]

On 04/06/2017 08:47 AM, Christoph Hellwig wrote:
> On Thu, Apr 06, 2017 at 08:33:55AM -0600, Jens Axboe wrote:
>> That is exactly what my recommendation was at lsfmm as well - fork
>> xfstest, prune bits we don't need, and off we go. I'll get around to
>> it soonish.
> 
> So you volunteer to do it?  I was -><- this close to offering it myself,
> and than thought of all the other backlog I have on my plate :)

Yeah, I'll get something cobbled together next week.

-- 
Jens Axboe

Re: [PATCH 0/6] RFC add blkdev tests v2

[Johannes Thumshirn]

On Thu, Apr 06, 2017 at 08:51:36AM -0600, Jens Axboe wrote:
> On 04/06/2017 08:47 AM, Christoph Hellwig wrote:
> > On Thu, Apr 06, 2017 at 08:33:55AM -0600, Jens Axboe wrote:
> >> That is exactly what my recommendation was at lsfmm as well - fork
> >> xfstest, prune bits we don't need, and off we go. I'll get around to
> >> it soonish.
> > 
> > So you volunteer to do it?  I was -><- this close to offering it myself,
> > and than thought of all the other backlog I have on my plate :)
> 
> Yeah, I'll get something cobbled together next week.

Yay!

But please keep the dependencies small. It'll suck when we'd have to pull in a
gazillion megabytes for perl/pyhthon/whatever.

Thanks,
	Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N�rnberg
GF: Felix Imend�rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N�rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [PATCH 0/6] RFC add blkdev tests v2

[Johannes Thumshirn]

On Thu, Apr 06, 2017 at 08:51:36AM -0600, Jens Axboe wrote:
> On 04/06/2017 08:47 AM, Christoph Hellwig wrote:
> > On Thu, Apr 06, 2017 at 08:33:55AM -0600, Jens Axboe wrote:
> >> That is exactly what my recommendation was at lsfmm as well - fork
> >> xfstest, prune bits we don't need, and off we go. I'll get around to
> >> it soonish.
> > 
> > So you volunteer to do it?  I was -><- this close to offering it myself,
> > and than thought of all the other backlog I have on my plate :)
> 
> Yeah, I'll get something cobbled together next week.

Yay!

But please keep the dependencies small. It'll suck when we'd have to pull in a
gazillion megabytes for perl/pyhthon/whatever.

Thanks,
	Johannes
-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [PATCH 0/6] RFC add blkdev tests v2

[Eric Sandeen]

On 4/6/17 10:20 AM, Johannes Thumshirn wrote:
> On Thu, Apr 06, 2017 at 08:51:36AM -0600, Jens Axboe wrote:
>> On 04/06/2017 08:47 AM, Christoph Hellwig wrote:
>>> On Thu, Apr 06, 2017 at 08:33:55AM -0600, Jens Axboe wrote:
>>>> That is exactly what my recommendation was at lsfmm as well - fork
>>>> xfstest, prune bits we don't need, and off we go. I'll get around to
>>>> it soonish.
>>>
>>> So you volunteer to do it?  I was -><- this close to offering it myself,
>>> and than thought of all the other backlog I have on my plate :)
>>
>> Yeah, I'll get something cobbled together next week.
> 
> Yay!
> 
> But please keep the dependencies small. It'll suck when we'd have to pull in a
> gazillion megabytes for perl/pyhthon/whatever.

Full-blown xfstests doesn't have /that/ much in the way of deps,
on debian it's ostensibly just:

                sudo apt-get install xfslibs-dev uuid-dev libtool-bin \
                e2fsprogs automake gcc libuuid1 quota attr libattr1-dev make \
                libacl1-dev libaio-dev xfsprogs libgdbm-dev gawk fio dbench \
                uuid-runtime

and lots of that might not be needed for non-fs tests.

Otherwise well, you do need bash ... ;)

-Eric

Re: [PATCH 0/6] RFC add blkdev tests v2

[Johannes Thumshirn]

On Thu, Apr 06, 2017 at 10:32:43AM -0500, Eric Sandeen wrote:
> Full-blown xfstests doesn't have /that/ much in the way of deps,
> on debian it's ostensibly just:
> 
>                 sudo apt-get install xfslibs-dev uuid-dev libtool-bin \
>                 e2fsprogs automake gcc libuuid1 quota attr libattr1-dev make \
>                 libacl1-dev libaio-dev xfsprogs libgdbm-dev gawk fio dbench \
>                 uuid-runtime
> 
> and lots of that might not be needed for non-fs tests.
> 
> Otherwise well, you do need bash ... ;)

Yeah that's OK. Maybe I misphrased it a bit. No more additional dependencies.
Currently xfstest is "small" engough to be run in an initrd and that's good
for qemu -kernel .. -initrd .. style test runs.

Byte,
	Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N�rnberg
GF: Felix Imend�rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N�rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [PATCH 0/6] RFC add blkdev tests v2

[Johannes Thumshirn]

On Thu, Apr 06, 2017 at 10:32:43AM -0500, Eric Sandeen wrote:
> Full-blown xfstests doesn't have /that/ much in the way of deps,
> on debian it's ostensibly just:
> 
>                 sudo apt-get install xfslibs-dev uuid-dev libtool-bin \
>                 e2fsprogs automake gcc libuuid1 quota attr libattr1-dev make \
>                 libacl1-dev libaio-dev xfsprogs libgdbm-dev gawk fio dbench \
>                 uuid-runtime
> 
> and lots of that might not be needed for non-fs tests.
> 
> Otherwise well, you do need bash ... ;)

Yeah that's OK. Maybe I misphrased it a bit. No more additional dependencies.
Currently xfstest is "small" engough to be run in an initrd and that's good
for qemu -kernel .. -initrd .. style test runs.

Byte,
	Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Re: [PATCH 0/6] RFC add blkdev tests v2

[Jens Axboe]

On 04/06/2017 12:15 PM, Johannes Thumshirn wrote:
> On Thu, Apr 06, 2017 at 10:32:43AM -0500, Eric Sandeen wrote:
>> Full-blown xfstests doesn't have /that/ much in the way of deps,
>> on debian it's ostensibly just:
>>
>>                 sudo apt-get install xfslibs-dev uuid-dev libtool-bin \
>>                 e2fsprogs automake gcc libuuid1 quota attr libattr1-dev make \
>>                 libacl1-dev libaio-dev xfsprogs libgdbm-dev gawk fio dbench \
>>                 uuid-runtime
>>
>> and lots of that might not be needed for non-fs tests.
>>
>> Otherwise well, you do need bash ... ;)
> 
> Yeah that's OK. Maybe I misphrased it a bit. No more additional dependencies.
> Currently xfstest is "small" engough to be run in an initrd and that's good
> for qemu -kernel .. -initrd .. style test runs.

We share that goal. The block related tests will have _fewer_ dependencies.
And it'll also be Linux only, so we don't have to worry about cross platform.

-- 
Jens Axboe

Re: [PATCH 0/6] RFC add blkdev tests v2

[Christoph Hellwig]

On Thu, Apr 06, 2017 at 12:19:46PM -0600, Jens Axboe wrote:
> We share that goal. The block related tests will have _fewer_ dependencies.
> And it'll also be Linux only, so we don't have to worry about cross platform.

xfstests effectively has been Linux-only for many years.  We've just
been to lazy to remove the remaining IRIX bits..