Re: [PATCH 0/4] Enable Fault Injection for RTRS

From: Chuck Lever III <chuck.lever@oracle.com>
To: Gioh Kim <gi-oh.kim@ionos.com>
Cc: linux-rdma <linux-rdma@vger.kernel.org>,
	"linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
	"bvanassche@acm.org" <bvanassche@acm.org>,
	"leon@kernel.org" <leon@kernel.org>,
	"dledford@redhat.com" <dledford@redhat.com>,
	"jgg@ziepe.ca" <jgg@ziepe.ca>,
	"haris.iqbal@ionos.com" <haris.iqbal@ionos.com>,
	"jinpu.wang@ionos.com" <jinpu.wang@ionos.com>,
	"akinobu.mita@gmail.com" <akinobu.mita@gmail.com>,
	"corbet@lwn.net" <corbet@lwn.net>
Subject: Re: [PATCH 0/4] Enable Fault Injection for RTRS
Date: Tue, 6 Apr 2021 14:20:09 +0000	[thread overview]
Message-ID: <6DBCD231-E108-4F4C-AFBF-688E988DACD7@oracle.com> (raw)
In-Reply-To: <20210406115049.196527-1-gi-oh.kim@ionos.com>

> On Apr 6, 2021, at 7:50 AM, Gioh Kim <gi-oh.kim@ionos.com> wrote:
> 
> My colleagues and I would like to apply the fault injection
> of the Linux to test error handling of RTRS module. RTRS module
> consists of client and server modules that are connected via
> Infiniband network. So it is important for the client to receive
> the error of the server and handle it smoothly.

I am a fan of fault injection. In fact I added a disconnect fault
injector for RPC that's in the kernel now, and it uses debugfs
as its control interface.

But that was years ago. If I were doing this today, I'd consider
kprobes, since fault injection is generally not something that
is consumed by users or administrators in a distributed kernel.

Have you considered injection via kprobes or eBPF instead of
adding permanent code?

> When debugfs is enabled, RTRS is able to export interfaces
> to fail RTRS client and server.
> Following fault injection points are enabled:
> - fail a request processing on RTRS client side
> - fail a heart-beat transferation on RTRS server side
> 
> This patch set is just a starting point. We will enable various
> faults and test as many error cases as possible.
> 
> Best regards
> 
> Gioh Kim (4):
>  RDMA/rtrs: Enable the fault-injection
>  RDMA/rtrs-clt: Inject a fault at request processing
>  RDMA/rtrs-srv: Inject a fault at heart-beat sending
>  docs: fault-injection: Add fault-injection manual of RTRS
> 
> .../fault-injection/rtrs-fault-injection.rst  | 83 +++++++++++++++++++
> drivers/infiniband/ulp/rtrs/Makefile          |  2 +
> drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c  | 44 ++++++++++
> drivers/infiniband/ulp/rtrs/rtrs-clt.c        |  7 ++
> drivers/infiniband/ulp/rtrs/rtrs-clt.h        | 13 +++
> drivers/infiniband/ulp/rtrs/rtrs-fault.c      | 52 ++++++++++++
> drivers/infiniband/ulp/rtrs/rtrs-fault.h      | 28 +++++++
> drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c  | 44 ++++++++++
> drivers/infiniband/ulp/rtrs/rtrs-srv.c        |  5 ++
> drivers/infiniband/ulp/rtrs/rtrs-srv.h        | 13 +++
> 10 files changed, 291 insertions(+)
> create mode 100644 Documentation/fault-injection/rtrs-fault-injection.rst
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-fault.c
> create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-fault.h
> 
> -- 
> 2.25.1
> 

--
Chuck Lever