Re: [PATCH 0/4] Enable Fault Injection for RTRS

From: Gioh Kim <gi-oh.kim@ionos.com>
To: Chuck Lever III <chuck.lever@oracle.com>
Cc: linux-rdma <linux-rdma@vger.kernel.org>,
	"linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
	"bvanassche@acm.org" <bvanassche@acm.org>,
	"leon@kernel.org" <leon@kernel.org>,
	"dledford@redhat.com" <dledford@redhat.com>,
	"jgg@ziepe.ca" <jgg@ziepe.ca>,
	"haris.iqbal@ionos.com" <haris.iqbal@ionos.com>,
	"jinpu.wang@ionos.com" <jinpu.wang@ionos.com>,
	"akinobu.mita@gmail.com" <akinobu.mita@gmail.com>,
	"corbet@lwn.net" <corbet@lwn.net>
Subject: Re: [PATCH 0/4] Enable Fault Injection for RTRS
Date: Tue, 6 Apr 2021 17:20:07 +0200	[thread overview]
Message-ID: <CAJX1YtYcGJkTH6CyV7uoudVuRamYUVeA6UbmQkEfYq+rxTH=Gw@mail.gmail.com> (raw)
In-Reply-To: <6DBCD231-E108-4F4C-AFBF-688E988DACD7@oracle.com>

On Tue, Apr 6, 2021 at 4:20 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
>
>
>
> > On Apr 6, 2021, at 7:50 AM, Gioh Kim <gi-oh.kim@ionos.com> wrote:
> >
> > My colleagues and I would like to apply the fault injection
> > of the Linux to test error handling of RTRS module. RTRS module
> > consists of client and server modules that are connected via
> > Infiniband network. So it is important for the client to receive
> > the error of the server and handle it smoothly.
>
> I am a fan of fault injection. In fact I added a disconnect fault
> injector for RPC that's in the kernel now, and it uses debugfs
> as its control interface.
>
> But that was years ago. If I were doing this today, I'd consider
> kprobes, since fault injection is generally not something that
> is consumed by users or administrators in a distributed kernel.
>
> Have you considered injection via kprobes or eBPF instead of
> adding permanent code?

I have not considered the eBPF yet.
I will have a discussion with my colleagues about that.
Thank you for the information.

>
>
> > When debugfs is enabled, RTRS is able to export interfaces
> > to fail RTRS client and server.
> > Following fault injection points are enabled:
> > - fail a request processing on RTRS client side
> > - fail a heart-beat transferation on RTRS server side
> >
> > This patch set is just a starting point. We will enable various
> > faults and test as many error cases as possible.
> >
> > Best regards
> >
> > Gioh Kim (4):
> >  RDMA/rtrs: Enable the fault-injection
> >  RDMA/rtrs-clt: Inject a fault at request processing
> >  RDMA/rtrs-srv: Inject a fault at heart-beat sending
> >  docs: fault-injection: Add fault-injection manual of RTRS
> >
> > .../fault-injection/rtrs-fault-injection.rst  | 83 +++++++++++++++++++
> > drivers/infiniband/ulp/rtrs/Makefile          |  2 +
> > drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c  | 44 ++++++++++
> > drivers/infiniband/ulp/rtrs/rtrs-clt.c        |  7 ++
> > drivers/infiniband/ulp/rtrs/rtrs-clt.h        | 13 +++
> > drivers/infiniband/ulp/rtrs/rtrs-fault.c      | 52 ++++++++++++
> > drivers/infiniband/ulp/rtrs/rtrs-fault.h      | 28 +++++++
> > drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c  | 44 ++++++++++
> > drivers/infiniband/ulp/rtrs/rtrs-srv.c        |  5 ++
> > drivers/infiniband/ulp/rtrs/rtrs-srv.h        | 13 +++
> > 10 files changed, 291 insertions(+)
> > create mode 100644 Documentation/fault-injection/rtrs-fault-injection.rst
> > create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-fault.c
> > create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-fault.h
> >
> > --
> > 2.25.1
> >
>
> --
> Chuck Lever
>
>
>