[PATCH 0/2][V2] io.latency test for blktests

[PATCH 0/2][V2] io.latency test for blktests

[Josef Bacik]

v1->v2:
- dropped my python library, TIL about jq.
- fixed the spelling mistakes in the test.

-- Original message --

This patchset is to add a test to verify io.latency is working properly, and to
add all the supporting code to run that test.

First is the cgroup2 infrastructure which is fairly straightforward.  Just
verifies we have cgroup2, and gives us the helpers to check and make sure we
have the right controllers in place for the test.

The second patch brings over some python scripts I put in xfstests for parsing
the fio json output.  I looked at the existing fio performance stuff in
blktests, but we only capture bw stuff, which is wonky with this particular test
because once the fast group is finished the slow group is allowed to go as fast
as it wants.  So I needed this to pull out actual jobtime spent.  This will give
us flexibility to pull out other fio performance data in the future.

The final patch is the test itself.  It simply runs a job by itself to get a
baseline view of the disk performance.  Then it creates 2 cgroups, one fast and
one slow, and runs the same job simultaneously in both groups.  The result
should be that the fast group takes just slightly longer time than the baseline
(I use a 15% threshold to be safe), and that the slow one takes considerably
longer.  Thanks,

Josef

[PATCH 1/2] blktests: add cgroup2 infrastructure

[Josef Bacik]

In order to test io.latency and other cgroup related things we need some
supporting helpers to setup and tear down cgroup2.  This adds support
for checking that we can even configure cgroup2 things, set them up if
need be, and then add the cleanup stuff to the main cleanup function so
everything is always in a clean state.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 check     |  2 ++
 common/rc | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+)

diff --git a/check b/check
index ebd87c097e25..1c9dbc518fa1 100755
--- a/check
+++ b/check
@@ -294,6 +294,8 @@ _cleanup() {
 		done
 		unset RESTORE_CPUS_ONLINE
 	fi
+
+	_cleanup_cgroup2
 }
 
 _call_test() {
diff --git a/common/rc b/common/rc
index 8a892bcd5fde..a785f2329687 100644
--- a/common/rc
+++ b/common/rc
@@ -202,3 +202,51 @@ _test_dev_in_hotplug_slot() {
 _filter_xfs_io_error() {
 	sed -e 's/^\(.*\)64\(: .*$\)/\1\2/'
 }
+
+_cgroup2_base_dir()
+{
+	grep cgroup2 /proc/mounts | awk '{ print $2 }'
+}
+
+_cleanup_cgroup2()
+{
+	_dir=$(_cgroup2_base_dir)/blktests
+	[ -d "${_dir}" ] || return
+
+	for i in $(find ${_dir} -type d | tac)
+	do
+		rmdir $i
+	done
+}
+
+_have_cgroup2()
+{
+	if ! grep -q 'cgroup2' /proc/mounts; then
+		SKIP_REASON="This test requires cgroup2"
+		return 1
+	fi
+	return 0
+}
+
+_have_cgroup2_controller_file()
+{
+	_have_cgroup2 || return 1
+
+	_controller=$1
+	_file=$2
+	_dir=$(_cgroup2_base_dir)
+
+	if ! grep -q ${_controller} ${_dir}/cgroup.controllers; then
+		SKIP_REASON="No support for ${_controller} cgroup controller"
+		return 1
+	fi
+
+	mkdir ${_dir}/blktests
+	echo "+${_controller}" > ${_dir}/cgroup.subtree_control
+	if [ ! -f ${_dir}/blktests/${_file} ]; then
+		_cleanup_cgroup2
+		SKIP_REASON="Cgroup file ${_file} doesn't exist"
+		return 1
+	fi
+	return 0
+}
-- 
2.14.3

[PATCH 2/2] blktests: block/025: an io.latency test

[Josef Bacik]

This is a test to verify io.latency is working properly.  It does this
by first running a fio job by itself to figure out how fast it runs.
Then we calculate some thresholds, set up 2 cgroups, a fast and a slow
cgroup, and then run the same job in both groups simultaneously.  We
should see the slow group get throttled until the first cgroup is able
to finish, and then the slow cgroup will be allowed to finish.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 tests/block/025     | 133 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/block/025.out |   1 +
 2 files changed, 134 insertions(+)
 create mode 100644 tests/block/025
 create mode 100644 tests/block/025.out

diff --git a/tests/block/025 b/tests/block/025
new file mode 100644
index 000000000000..cdaa35c5e335
--- /dev/null
+++ b/tests/block/025
@@ -0,0 +1,133 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+#
+# Test io.latency to make sure it's protecting the higher priority group
+# properly.
+
+. tests/block/rc
+
+DESCRIPTION="test the io.latency interface to make sure it's working right"
+
+requires() {
+	_have_cgroup2_controller_file io io.latency && _have_fio && \
+		_have_program jq
+}
+
+_fio_results_key() {
+	_job=$1
+	_key=$2
+	_resultfile=$3
+
+	jq '.jobs[] | select(.jobname == "'${_job}'") | .'${_key} ${_resultfile}
+}
+
+test_device() {
+	local fio_config_single fio_config_double fio_results fio_args qd
+
+	fio_config_single=${TMPDIR}/single.fio
+	fio_config_double=${TMPDIR}/double.fio
+	fio_results=${TMPDIR}/025.json
+	fio_args="--output-format=json --output=${fio_results}"
+	qd=$(cat ${TEST_DEV_SYSFS}/queue/nr_requests)
+
+	cat << EOF > "${fio_config_single}"
+	[fast]
+	filename=${TEST_DEV}
+	direct=1
+	allrandrepeat=1
+	readwrite=randrw
+	size=4G
+	ioengine=libaio
+	iodepth=${qd}
+	fallocate=none
+	randseed=12345
+EOF
+
+	cat << EOF > "${fio_config_double}"
+	[global]
+	filename=${TEST_DEV}
+	direct=1
+	allrandrepeat=1
+	readwrite=randrw
+	size=4G
+	ioengine=libaio
+	iodepth=${qd}
+	fallocate=none
+	randseed=12345
+
+	[fast]
+	cgroup=blktests/fast
+
+	[slow]
+	cgroup=blktests/slow
+EOF
+	# We run the test once so we have an idea of how fast this workload will
+	# go with nobody else doing IO on the device.
+	if ! fio ${fio_args} ${fio_config_single}; then
+		echo "fio exited with status $?"
+		return 1
+	fi
+
+	_time_taken=$(_fio_results_key fast job_runtime ${fio_results})
+	if [ "${_time_taken}" = "" ]; then
+		echo "fio doesn't report job_runtime"
+		return 1
+	fi
+
+	echo "normal time taken ${_time_taken}" >> $FULL
+
+	# There's no way to predict how the two workloads are going to affect
+	# each other, so we weant to set thresholds to something reasonable so
+	# we can verify io.latency is doing something.  This means we set 15%
+	# for the fast cgroup, just to give us enough wiggle room as throttling
+	# doesn't happen immediately.  But if we have a super fast disk we could
+	# run both groups really fast and make it under our fast threshold, so
+	# we need to set a threshold for the slow group at 50%.  We assume that
+	# if it was faster than 50% of the fast threshold then we probably
+	# didn't throttle and we can assume io.latency is broken.
+	_fast_thresh=$((${_time_taken} + ${_time_taken} * 15 / 100))
+	_slow_thresh=$((${_time_taken} + ${_time_taken} * 50 / 100))
+	echo "fast threshold time is ${_fast_thresh}" >> $FULL
+	echo "slow threshold time is ${_slow_thresh}" >> $FULL
+
+	# Create the cgroup files
+	_dir=$(_cgroup2_base_dir)/blktests
+	echo "+io" > ${_dir}/cgroup.subtree_control
+	mkdir ${_dir}/fast
+	mkdir ${_dir}/slow
+
+	# We set the target to 1usec because we could have a fast device that is
+	# capable of remarkable IO latencies that would skew the test.  It needs
+	# to be low enough that we do actually throttle the slow group,
+	# otherwise the test will fail when there's nothing wrong.
+	_major=$((0x$(stat -c "%t" ${TEST_DEV})))
+	_minor=$((0x$(stat -c "%T" ${TEST_DEV})))
+	echo "${_major}:${_minor} is our device" >> $FULL
+	if ! echo "${_major}:${_minor} target=1" > ${_dir}/fast/io.latency; then
+		echo "Failed to set our latency target"
+		return 1
+	fi
+
+	if ! fio ${fio_args} ${fio_config_double}; then
+		echo "fio exited with status $?"
+		return 1
+	fi
+
+	_fast_time=$(_fio_results_key fast job_runtime ${fio_results})
+	echo "Fast time ${_fast_time}" >> $FULL
+	_slow_time=$(_fio_results_key slow job_runtime ${fio_results})
+	echo "Slow time ${_slow_time}" >> $FULL
+
+	if [ ${_fast_thresh} -lt ${_fast_time} ]; then
+		echo "Too much of a performance drop for the protected workload"
+		return 1
+	fi
+
+	if [ ${_slow_thresh} -gt ${_slow_time} ]; then
+		echo "The slow group does not appear to have been throttled"
+		return 1
+	fi
+
+	echo "Silence is golden"
+	return 0
+}
diff --git a/tests/block/025.out b/tests/block/025.out
new file mode 100644
index 000000000000..c19ff631b9b5
--- /dev/null
+++ b/tests/block/025.out
@@ -0,0 +1 @@
+Silence is golden
-- 
2.14.3

Re: [PATCH 0/2][V2] io.latency test for blktests

[Omar Sandoval]

On Wed, Dec 05, 2018 at 10:34:02AM -0500, Josef Bacik wrote:
> v1->v2:
> - dropped my python library, TIL about jq.
> - fixed the spelling mistakes in the test.
> 
> -- Original message --
> 
> This patchset is to add a test to verify io.latency is working properly, and to
> add all the supporting code to run that test.
> 
> First is the cgroup2 infrastructure which is fairly straightforward.  Just
> verifies we have cgroup2, and gives us the helpers to check and make sure we
> have the right controllers in place for the test.
> 
> The second patch brings over some python scripts I put in xfstests for parsing
> the fio json output.  I looked at the existing fio performance stuff in
> blktests, but we only capture bw stuff, which is wonky with this particular test
> because once the fast group is finished the slow group is allowed to go as fast
> as it wants.  So I needed this to pull out actual jobtime spent.  This will give
> us flexibility to pull out other fio performance data in the future.
> 
> The final patch is the test itself.  It simply runs a job by itself to get a
> baseline view of the disk performance.  Then it creates 2 cgroups, one fast and
> one slow, and runs the same job simultaneously in both groups.  The result
> should be that the fast group takes just slightly longer time than the baseline
> (I use a 15% threshold to be safe), and that the slow one takes considerably
> longer.  Thanks,

I cleaned up a ton of shellcheck warnings (from `make check`) and pushed
to https://github.com/osandov/blktests/tree/josef. On I tested with QEMU
on Jens' for-next branch. With an emulated NVMe device, it failed with
"Too much of a performance drop for the protected workload". On
virtio-blk, I hit this:

[ 1843.056452] INFO: task fio:20750 blocked for more than 120 seconds.
[ 1843.057495]       Not tainted 4.20.0-rc5-00251-g90efb26fa9a4 #19
[ 1843.058487] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1843.059769] fio             D    0 20750  20747 0x00000080
[ 1843.060688] Call Trace:
[ 1843.061123]  ? __schedule+0x286/0x870
[ 1843.061735]  ? blkcg_iolatency_done_bio+0x680/0x680
[ 1843.062574]  ? blkcg_iolatency_cleanup+0x60/0x60
[ 1843.063347]  schedule+0x32/0x80
[ 1843.063874]  io_schedule+0x12/0x40
[ 1843.064449]  rq_qos_wait+0x9a/0x120
[ 1843.065007]  ? karma_partition+0x210/0x210
[ 1843.065661]  ? blkcg_iolatency_done_bio+0x680/0x680
[ 1843.066435]  blkcg_iolatency_throttle+0x185/0x360
[ 1843.067196]  __rq_qos_throttle+0x23/0x30
[ 1843.067958]  blk_mq_make_request+0x101/0x5c0
[ 1843.068637]  generic_make_request+0x1b3/0x3c0
[ 1843.069329]  submit_bio+0x45/0x140
[ 1843.069876]  blkdev_direct_IO+0x3db/0x440
[ 1843.070527]  ? aio_complete+0x2f0/0x2f0
[ 1843.071146]  generic_file_direct_write+0x96/0x160
[ 1843.071880]  __generic_file_write_iter+0xb3/0x1c0
[ 1843.072599]  ? blk_mq_dispatch_rq_list+0x3aa/0x550
[ 1843.073340]  blkdev_write_iter+0xa0/0x120
[ 1843.073960]  ? __fget+0x6e/0xa0
[ 1843.074452]  aio_write+0x11f/0x1d0
[ 1843.074979]  ? __blk_mq_run_hw_queue+0x6f/0xe0
[ 1843.075658]  ? __check_object_size+0xa0/0x189
[ 1843.076345]  ? preempt_count_add+0x5a/0xb0
[ 1843.077086]  ? aio_read_events+0x259/0x380
[ 1843.077819]  ? kmem_cache_alloc+0x16e/0x1c0
[ 1843.078427]  io_submit_one+0x4a8/0x790
[ 1843.078975]  ? read_events+0x76/0x150
[ 1843.079510]  __se_sys_io_submit+0x98/0x1a0
[ 1843.080116]  ? syscall_trace_enter+0x1d3/0x2d0
[ 1843.080785]  do_syscall_64+0x55/0x160
[ 1843.081404]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1843.082210] RIP: 0033:0x7f6e571fc4ed
[ 1843.082763] Code: Bad RIP value.
[ 1843.083268] RSP: 002b:00007ffc212b76f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d1
[ 1843.084445] RAX: ffffffffffffffda RBX: 00007f6e4c876870 RCX: 00007f6e571fc4ed
[ 1843.085545] RDX: 0000557c4bc11208 RSI: 0000000000000001 RDI: 00007f6e4c85e000
[ 1843.086251] RBP: 00007f6e4c85e000 R08: 0000557c4bc2b130 R09: 00000000000002f8
[ 1843.087308] R10: 0000557c4bbf4470 R11: 0000000000000246 R12: 0000000000000001
[ 1843.088310] R13: 0000000000000000 R14: 0000557c4bc11208 R15: 00007f6e2b17f070

Re: [PATCH 1/2] blktests: add cgroup2 infrastructure

[Omar Sandoval]

On Wed, Dec 05, 2018 at 10:34:03AM -0500, Josef Bacik wrote:
> In order to test io.latency and other cgroup related things we need some
> supporting helpers to setup and tear down cgroup2.  This adds support
> for checking that we can even configure cgroup2 things, set them up if
> need be, and then add the cleanup stuff to the main cleanup function so
> everything is always in a clean state.
> 
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>

Since Dennis' test needed this, I merged this patch with the fixup for
check folded in.