Re: Encountering errors while using RNBD over rxe for v5.14

[Haris Iqbal <haris.iqbal@ionos.com>]

Hi Guoqing,

On Fri, Apr 29, 2022 at 10:26 AM Guoqing Jiang <guoqing.jiang@linux.dev> wrote:
>
> Hi Haris,
>
> On 4/20/22 6:28 PM, Haris Iqbal wrote:
> > Hello,
> >
> > We are facing some issues with the rxe driver in v5.14 (tested with 5.14.21)
> >
> > After mapping a single RNBD device with 2 rxe interfaces, and with the
> > below fio config,
> >
> > [global]
> > description=Emulation of Storage Server Access Pattern
> > bssplit=512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4
> > fadvise_hint=0
> > rw=randrw:2
> > direct=1
> > random_distribution=zipf:1.2
> > time_based=1
> > runtime=60
> > ramp_time=1
> > ioengine=libaio
> > iodepth=128
> > iodepth_batch_submit=128
> > iodepth_batch_complete_min=1
> > iodepth_batch_complete_max=128
> > numjobs=1
> > group_reporting
> >
> > [job1]
> > filename=/dev/rnbd0
> >
> >
> > We observe the following error,
> >
> > [Fri Mar 25 19:08:03 2022] rtrs_client L353: <blya>: Failed
> > IB_WR_LOCAL_INV: WR flushed
> > [Fri Mar 25 19:08:03 2022] rtrs_client L333: <blya>: Failed
> > IB_WR_REG_MR: WR flushed
> > [Fri Mar 25 19:08:03 2022] rtrs_client L333: <blya>: Failed
> > IB_WR_REG_MR: WR flushed
> > [Fri Mar 25 19:08:34 2022] rtrs_client L353: <blya>: Failed
> > IB_WR_LOCAL_INV: WR flushed
> > [Fri Mar 25 19:08:34 2022] rtrs_client L353: <blya>:*Failed IB_WR_LOCAL_INV: WR flushed*
>
> I got similar message but I am not certain it is the same one, pls see
> the previous report,
>
> https://lore.kernel.org/linux-rdma/20220210073655.42281-1-guoqing.jiang@linux.dev/

I went through the emails, and the error looks similar but I am not
sure if its related. The change to write_lock_bh was done after v5.14
I think, and the code where I am seeing this error still has irq
versions of those lock (e.g. write_lock_irqsave).

I don't have much knowledge of rxe code, hence I cannot really say for sure.

Thanks for the response.

>
> Thanks,
> Guoqing