* [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests
@ 2018-12-20 18:18 Dennis Zhou
2018-12-20 18:18 ` [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check Dennis Zhou
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Dennis Zhou @ 2018-12-20 18:18 UTC (permalink / raw)
To: Omar Sandoval, Ming Lei; +Cc: kernel-team, linux-block, Dennis Zhou
Hi,
v2:
There was a minor typo in 0001 pointed out by Omar which I fixed here.
0002 was refactored to use common/scsi_debug. I also rebased onto the
top of osandov#josef changing the test from block/022 -> block/027.
From v1:
Ming Lei's scsi-stress-remove test found a bug in blkg destruction [1]
where bios being created when the request_queue was being cleaned up
threw a NPE in blkg association. The fix is currently being discussed in
[2]. To make this test more accessible, I've ported it to blktests with
Ming Lei's copyright. I've tested this in my qemu instance and verified
we do not see the NPE on a fixed kernel.
Ming, please let me know if you have any objections.
[1] https://lore.kernel.org/linux-block/CACVXFVO_QXipD3cmPvpLyBYSiEcWPN_ThQ=0pO9AwLqN-Lv93w@mail.gmail.com
[2] https://lore.kernel.org/lkml/20181211230308.66276-1-dennis@kernel.org/
This patchset is ontop of osandov#josef 98db2e11c97f.
diffstats below:
Dennis Zhou (2):
blktests: split out cgroup2 controller and file check
blktests: add Ming Lei's scsi-stress-remove
common/cgroup | 18 +++++++----
tests/block/027 | 73 +++++++++++++++++++++++++++++++++++++++++++++
tests/block/027.out | 2 ++
3 files changed, 88 insertions(+), 5 deletions(-)
create mode 100755 tests/block/027
create mode 100644 tests/block/027.out
Thanks,
Dennis
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check
2018-12-20 18:18 [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests Dennis Zhou
@ 2018-12-20 18:18 ` Dennis Zhou
2018-12-20 18:18 ` [PATCH blktests 2/2] blktests: add Ming Lei's scsi-stress-remove Dennis Zhou
2018-12-20 23:56 ` [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests Omar Sandoval
2 siblings, 0 replies; 6+ messages in thread
From: Dennis Zhou @ 2018-12-20 18:18 UTC (permalink / raw)
To: Omar Sandoval, Ming Lei; +Cc: kernel-team, linux-block, Dennis Zhou
This is a prep patch for a new test that will race blkg association and
request_queue cleanup. As blkg association is a underlying cgroup io
controller feature, we need the ability to check if the controller is
available.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
---
v2:
- fixed minor typo.
common/cgroup | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/common/cgroup b/common/cgroup
index d445093..48e546f 100644
--- a/common/cgroup
+++ b/common/cgroup
@@ -37,19 +37,27 @@ _have_cgroup2()
return 0
}
-_have_cgroup2_controller_file()
+_have_cgroup2_controller()
{
- _have_cgroup2 || return 1
-
local controller="$1"
- local file="$2"
- local dir
+
+ _have_cgroup2 || return 1
dir="$(_cgroup2_base_dir)"
+
if ! grep -q "$controller" "$dir/cgroup.controllers"; then
SKIP_REASON="no support for $controller cgroup controller; if it is enabled, you may need to boot with cgroup_no_v1=$controller"
return 1
fi
+}
+
+_have_cgroup2_controller_file()
+{
+ local controller="$1"
+ local file="$2"
+ local dir
+
+ _have_cgroup2_controller "$controller" || return 1
mkdir "$dir/blktests"
echo "+$controller" > "$dir/cgroup.subtree_control"
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH blktests 2/2] blktests: add Ming Lei's scsi-stress-remove
2018-12-20 18:18 [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests Dennis Zhou
2018-12-20 18:18 ` [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check Dennis Zhou
@ 2018-12-20 18:18 ` Dennis Zhou
2018-12-20 23:56 ` [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests Omar Sandoval
2 siblings, 0 replies; 6+ messages in thread
From: Dennis Zhou @ 2018-12-20 18:18 UTC (permalink / raw)
To: Omar Sandoval, Ming Lei; +Cc: kernel-team, linux-block, Dennis Zhou
This test exposed a race condiiton when shutting down a request_queue
with active IO against it and blkg association for the IOs [1]. The
issue ended up being that while the request_queue will just start
failing requests, blkg destruction sets the q->root_blkg to %NULL. This
caused a NPE. This was fixed in [2].
So to help prevent this from happening again, integrate Ming's test into
blktests so that it can more easily be ran. Here I've ported it to fit
better into the blktests framework.
[1] https://lore.kernel.org/lkml/20181205171039.73066-1-dennis@kernel.org/
[2] https://lore.kernel.org/lkml/20181211230308.66276-1-dennis@kernel.org/
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Cc: Ming Lei <ming.lei@redhat.com>
---
v2:
- Used helpers in common/scsi_debug per Omar's suggestion.
tests/block/027 | 73 +++++++++++++++++++++++++++++++++++++++++++++
tests/block/027.out | 2 ++
2 files changed, 75 insertions(+)
create mode 100755 tests/block/027
create mode 100644 tests/block/027.out
diff --git a/tests/block/027 b/tests/block/027
new file mode 100755
index 0000000..d5e1463
--- /dev/null
+++ b/tests/block/027
@@ -0,0 +1,73 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2018 Ming Lei
+#
+# Regression test for commit 0273ac349f08 ("blkcg: handle dying request_queue
+# when associating a blkg")
+#
+# This tries to expose the race condition between blkg association and
+# request_queue shutdown. When a request_queue is shutdown, the corresponding
+# blkgs are destroyed. Any further associations should fail gracefully and not
+# cause a kernel panic.
+
+. tests/block/rc
+. common/scsi_debug
+. common/cgroup
+
+DESCRIPTION="test graceful shutdown of scsi_debug devices with running fio jobs"
+QUICK=1
+
+requires() {
+ _have_cgroup2_controller io && _have_scsi_debug && _have_fio
+}
+
+scsi_debug_stress_remove() {
+ if ! _init_scsi_debug "$@"; then
+ return
+ fi
+
+ # set higher aio limit
+ echo 524288 > /proc/sys/fs/aio-max-nr
+
+ local dev fio_jobs scheds
+ local cnt=1
+ for dev in "${SCSI_DEBUG_DEVICES[@]}"; do
+ # create fio job
+ fio_jobs=$fio_jobs" --name=job1 --filename=/dev/$dev "
+
+ # set queue scheduler and queue_depth
+ queue_path=/sys/block/$dev/queue
+ scheds=($(sed 's/[][]//g' "$queue_path/scheduler"))
+ sched_idx=$((cnt % ${#scheds[@]}))
+ echo "${scheds[$sched_idx]}" > "$queue_path/scheduler"
+ echo $cnt > "$queue_path/../device/queue_depth"
+ cnt=$((cnt+1))
+ done
+
+ local num_jobs=4 runtime=21
+ fio --rw=randread --size=128G --direct=1 --ioengine=libaio \
+ --iodepth=2048 --numjobs=$num_jobs --bs=4k \
+ --group_reporting=1 --group_reporting=1 --runtime=$runtime \
+ --loops=10000 "$fio_jobs" > "$FULL" 2>&1 &
+
+ sleep 7
+ local device_path
+ for dev in "${SCSI_DEBUG_DEVICES[@]}"; do
+ # shutdown devices in progress
+ device_path=/sys/block/$dev/device
+ [ -f "$device_path/delete" ] && echo 1 > "$device_path/delete"
+ done
+
+ wait
+
+ _exit_scsi_debug
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ scsi_debug_stress_remove virtual_gb=128 max_luns=21 ndelay=10000 \
+ max_queue=110
+
+ echo "Test complete"
+}
diff --git a/tests/block/027.out b/tests/block/027.out
new file mode 100644
index 0000000..b03498f
--- /dev/null
+++ b/tests/block/027.out
@@ -0,0 +1,2 @@
+Running block/027
+Test complete
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests
2018-12-20 18:18 [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests Dennis Zhou
2018-12-20 18:18 ` [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check Dennis Zhou
2018-12-20 18:18 ` [PATCH blktests 2/2] blktests: add Ming Lei's scsi-stress-remove Dennis Zhou
@ 2018-12-20 23:56 ` Omar Sandoval
2 siblings, 0 replies; 6+ messages in thread
From: Omar Sandoval @ 2018-12-20 23:56 UTC (permalink / raw)
To: Dennis Zhou; +Cc: Omar Sandoval, Ming Lei, kernel-team, linux-block
On Thu, Dec 20, 2018 at 12:18:24PM -0600, Dennis Zhou wrote:
> Hi,
>
> v2:
> There was a minor typo in 0001 pointed out by Omar which I fixed here.
> 0002 was refactored to use common/scsi_debug. I also rebased onto the
> top of osandov#josef changing the test from block/022 -> block/027.
>
> From v1:
> Ming Lei's scsi-stress-remove test found a bug in blkg destruction [1]
> where bios being created when the request_queue was being cleaned up
> threw a NPE in blkg association. The fix is currently being discussed in
> [2]. To make this test more accessible, I've ported it to blktests with
> Ming Lei's copyright. I've tested this in my qemu instance and verified
> we do not see the NPE on a fixed kernel.
>
> Ming, please let me know if you have any objections.
>
> [1] https://lore.kernel.org/linux-block/CACVXFVO_QXipD3cmPvpLyBYSiEcWPN_ThQ=0pO9AwLqN-Lv93w@mail.gmail.com
> [2] https://lore.kernel.org/lkml/20181211230308.66276-1-dennis@kernel.org/
>
> This patchset is ontop of osandov#josef 98db2e11c97f.
>
> diffstats below:
>
> Dennis Zhou (2):
> blktests: split out cgroup2 controller and file check
> blktests: add Ming Lei's scsi-stress-remove
>
> common/cgroup | 18 +++++++----
> tests/block/027 | 73 +++++++++++++++++++++++++++++++++++++++++++++
> tests/block/027.out | 2 ++
> 3 files changed, 88 insertions(+), 5 deletions(-)
> create mode 100755 tests/block/027
> create mode 100644 tests/block/027.out
Thanks, Dennis, applied along with Josef's cgroup infrastructure patch.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check
2018-12-12 23:09 ` [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check Dennis Zhou
@ 2018-12-19 18:34 ` Omar Sandoval
0 siblings, 0 replies; 6+ messages in thread
From: Omar Sandoval @ 2018-12-19 18:34 UTC (permalink / raw)
To: Dennis Zhou; +Cc: Omar Sandoval, Ming Lei, kernel-team, linux-block
On Wed, Dec 12, 2018 at 06:09:58PM -0500, Dennis Zhou wrote:
> This is a prep patch for a new test that will race blkg association and
> request_queue cleanup. As blkg association is a underlying cgroup io
> controller feature, we need the ability to check if the controller is
> available.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> ---
> common/cgroup | 18 +++++++++++++-----
> 1 file changed, 13 insertions(+), 5 deletions(-)
>
> diff --git a/common/cgroup b/common/cgroup
> index d445093..3481458 100644
> --- a/common/cgroup
> +++ b/common/cgroup
> @@ -37,19 +37,27 @@ _have_cgroup2()
> return 0
> }
>
> -_have_cgroup2_controller_file()
> +_have_cgroup2_controller()
> {
> - _have_cgroup2 || return 1
> -
> local controller="$1"
> - local file="$2"
> - local dir
> +
> + _have_cgroup2 || return 1
>
> dir="$(_cgroup2_base_dir)"
> +
> if ! grep -q "$controller" "$dir/cgroup.controllers"; then
> SKIP_REASON="no support for $controller cgroup controller; if it is enabled, you may need to boot with cgroup_no_v1=$controller"
> return 1
> fi
> +}
> +
> +_have_cgroup2_controller_file()
> +{
> + local controller="$1"
> + local file="$2"
> + local dir
> +
> + _have_cgroup_2_controller "$controller" || return 1
This should be _have_cgroup2_controller. I'll fix it when I apply it.
>
> mkdir "$dir/blktests"
> echo "+$controller" > "$dir/cgroup.subtree_control"
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check
2018-12-12 23:09 [PATCH blktests " Dennis Zhou
@ 2018-12-12 23:09 ` Dennis Zhou
2018-12-19 18:34 ` Omar Sandoval
0 siblings, 1 reply; 6+ messages in thread
From: Dennis Zhou @ 2018-12-12 23:09 UTC (permalink / raw)
To: Omar Sandoval, Ming Lei; +Cc: kernel-team, linux-block, Dennis Zhou
This is a prep patch for a new test that will race blkg association and
request_queue cleanup. As blkg association is a underlying cgroup io
controller feature, we need the ability to check if the controller is
available.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
---
common/cgroup | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/common/cgroup b/common/cgroup
index d445093..3481458 100644
--- a/common/cgroup
+++ b/common/cgroup
@@ -37,19 +37,27 @@ _have_cgroup2()
return 0
}
-_have_cgroup2_controller_file()
+_have_cgroup2_controller()
{
- _have_cgroup2 || return 1
-
local controller="$1"
- local file="$2"
- local dir
+
+ _have_cgroup2 || return 1
dir="$(_cgroup2_base_dir)"
+
if ! grep -q "$controller" "$dir/cgroup.controllers"; then
SKIP_REASON="no support for $controller cgroup controller; if it is enabled, you may need to boot with cgroup_no_v1=$controller"
return 1
fi
+}
+
+_have_cgroup2_controller_file()
+{
+ local controller="$1"
+ local file="$2"
+ local dir
+
+ _have_cgroup_2_controller "$controller" || return 1
mkdir "$dir/blktests"
echo "+$controller" > "$dir/cgroup.subtree_control"
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-12-20 23:56 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-20 18:18 [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests Dennis Zhou
2018-12-20 18:18 ` [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check Dennis Zhou
2018-12-20 18:18 ` [PATCH blktests 2/2] blktests: add Ming Lei's scsi-stress-remove Dennis Zhou
2018-12-20 23:56 ` [PATCH blktests v2 0/2] Add scsi-stress-remove to blktests Omar Sandoval
-- strict thread matches above, loose matches on Subject: below --
2018-12-12 23:09 [PATCH blktests " Dennis Zhou
2018-12-12 23:09 ` [PATCH blktests 1/2] blktests: split out cgroup2 controller and file check Dennis Zhou
2018-12-19 18:34 ` Omar Sandoval
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).