From: Gioh Kim <gi-oh.kim@ionos.com>
To: linux-rdma@vger.kernel.org, linux-doc@vger.kernel.org
Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com,
jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com,
akinobu.mita@gmail.com, corbet@lwn.net,
Gioh Kim <gi-oh.kim@cloud.ionos.com>
Subject: [PATCH 4/4] docs: fault-injection: Add fault-injection manual of RTRS
Date: Tue, 6 Apr 2021 13:50:49 +0200 [thread overview]
Message-ID: <20210406115049.196527-5-gi-oh.kim@ionos.com> (raw)
In-Reply-To: <20210406115049.196527-1-gi-oh.kim@ionos.com>
From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
It describes how to use the fault-injection of RTRS.
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
---
.../fault-injection/rtrs-fault-injection.rst | 83 +++++++++++++++++++
1 file changed, 83 insertions(+)
create mode 100644 Documentation/fault-injection/rtrs-fault-injection.rst
diff --git a/Documentation/fault-injection/rtrs-fault-injection.rst b/Documentation/fault-injection/rtrs-fault-injection.rst
new file mode 100644
index 000000000000..463869877a85
--- /dev/null
+++ b/Documentation/fault-injection/rtrs-fault-injection.rst
@@ -0,0 +1,83 @@
+RTRS (RDMA Transport) Fault Injection
+=====================================
+This document introduces how to enable and use the error injection of RTRS
+via debugfs in the /sys/kernel/debug directory. When enabled, users can
+enable specific error injection point and change the default status code
+via the debugfs.
+
+Following examples show how to inject an error into the RTRS.
+
+First, enable CONFIG_FAULT_INJECTION_DEBUG_FS kernel config,
+recompile the kernel. After booting up the kernel, map a target device.
+
+After mapping, /sys/kernel/debug/<session-name> directory is created
+on both of the client and the server.
+
+Example 1: Inject an error into request processing of rtrs-client
+-----------------------------------------------------------------
+
+Generate an error on one session of rtrs-client::
+
+ echo 100 > /sys/kernel/debug/ip\:192.168.123.144@ip\:192.168.123.190/fault_inject/probability
+ echo 1 > /sys/kernel/debug/ip\:192.168.123.144@ip\:192.168.123.190/fault_inject/times
+ echo 1 > /sys/kernel/debug/ip\:192.168.123.144@ip\:192.168.123.190/fault_inject/fail-request
+ dd if=/dev/rnbd0 of=./dd bs=1k count=10
+
+Expected Result::
+
+ dd succeeds but generates an IO error
+
+Message from dmesg::
+
+ FAULT_INJECTION: forcing a failure.
+ name fault_inject, interval 1, probability 100, space 0, times 1
+ CPU: 0 PID: 799 Comm: dd Tainted: G O 5.4.77-pserver+ #169
+ Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
+ Call Trace:
+ dump_stack+0x97/0xe0
+ should_fail.cold+0x5/0x11
+ rtrs_clt_should_fail_request+0x2f/0x50 [rtrs_client]
+ rtrs_clt_request+0x223/0x540 [rtrs_client]
+ rnbd_queue_rq+0x347/0x800 [rnbd_client]
+ __blk_mq_try_issue_directly+0x268/0x380
+ blk_mq_request_issue_directly+0x9a/0xe0
+ blk_mq_try_issue_list_directly+0xa3/0x170
+ blk_mq_sched_insert_requests+0x1de/0x340
+ blk_mq_flush_plug_list+0x488/0x620
+ blk_flush_plug_list+0x20f/0x250
+ blk_finish_plug+0x3c/0x54
+ read_pages+0x104/0x2b0
+ __do_page_cache_readahead+0x28b/0x2b0
+ ondemand_readahead+0x2cc/0x610
+ generic_file_read_iter+0xde0/0x11f0
+ new_sync_read+0x246/0x360
+ vfs_read+0xc1/0x1b0
+ ksys_read+0xc3/0x160
+ do_syscall_64+0x68/0x260
+ entry_SYSCALL_64_after_hwframe+0x49/0xbe
+ RIP: 0033:0x7f7ff4296461
+ Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8 e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54 49 89 d4 55 48
+ RSP: 002b:00007fffdceca5b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
+ RAX: ffffffffffffffda RBX: 000055c5eab6e3e0 RCX: 00007f7ff4296461
+ RDX: 0000000000000400 RSI: 000055c5ead27000 RDI: 0000000000000000
+ RBP: 0000000000000400 R08: 0000000000000003 R09: 00007f7ff4368260
+ R10: ffffffffffffff3b R11: 0000000000000246 R12: 000055c5ead27000
+ R13: 0000000000000000 R14: 0000000000000000 R15: 000055c5ead27000
+
+Example 2: rtrs-server does not send ACK to the heart-beat of rtrs-client
+-------------------------------------------------------------------------
+
+::
+
+ echo 100 > /sys/kernel/debug/ip\:192.168.123.190@ip\:192.168.123.144/fault_inject/probability
+ echo 5 > /sys/kernel/debug/ip\:192.168.123.190@ip\:192.168.123.144/fault_inject/times
+ echo 1 > /sys/kernel/debug/ip\:192.168.123.190@ip\:192.168.123.144/fault_inject/fail-hb-ack
+
+Expected Result::
+
+ If rtrs-server does not send ACK more than 5 times, rtrs-client tries reconnection.
+
+Check how many times rtrs-client did reconnection::
+
+ cat /sys/devices/virtual/rtrs-client/bla/paths/ip\:192.168.122.142@ip\:192.168.122.130/stats/reconnects
+ 1 0
--
2.25.1
next prev parent reply other threads:[~2021-04-06 11:51 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-06 11:50 [PATCH 0/4] Enable Fault Injection for RTRS Gioh Kim
2021-04-06 11:50 ` [PATCH 1/4] RDMA/rtrs: Enable the fault-injection Gioh Kim
2021-04-06 11:50 ` [PATCH 2/4] RDMA/rtrs-clt: Inject a fault at request processing Gioh Kim
2021-04-06 11:50 ` [PATCH 3/4] RDMA/rtrs-srv: Inject a fault at heart-beat sending Gioh Kim
2021-04-06 11:50 ` Gioh Kim [this message]
2021-04-06 14:20 ` [PATCH 0/4] Enable Fault Injection for RTRS Chuck Lever III
2021-04-06 15:20 ` Gioh Kim
2021-04-13 22:53 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210406115049.196527-5-gi-oh.kim@ionos.com \
--to=gi-oh.kim@ionos.com \
--cc=akinobu.mita@gmail.com \
--cc=bvanassche@acm.org \
--cc=corbet@lwn.net \
--cc=dledford@redhat.com \
--cc=gi-oh.kim@cloud.ionos.com \
--cc=haris.iqbal@ionos.com \
--cc=jgg@ziepe.ca \
--cc=jinpu.wang@ionos.com \
--cc=leon@kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).