question about the completion tasklet in the rxe driver

question about the completion tasklet in the rxe driver

[Bob Pearson]

I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the
three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion
handler. This has been in there since the driver went upstream so the history of why it is there is lost.

I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver
does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some
reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion
handler can't sleep anyway.

As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is
slightly better with the completion handler called inline rather than deferred to another tasklet.
If we can eliminate this there won't be anymore tasklets in the rxe driver.

Does anyone know why the tasklet was put in in the first place?

Thanks,

Bob

RE: question about the completion tasklet in the rxe driver

[Bernard Metzler]

> -----Original Message-----
> From: Bob Pearson <rpearsonhpe@gmail.com>
> Sent: Wednesday, 15 March 2023 22:56
> To: Jason Gunthorpe <jgg@nvidia.com>; Zhu Yanjun <zyjzyj2000@gmail.com>;
> linux-rdma@vger.kernel.org; Bernard Metzler <BMT@zurich.ibm.com>
> Subject: [EXTERNAL] question about the completion tasklet in the rxe driver
> 
> I have a goal of trying to get rid of all the tasklets in the rxe driver
> and with the replacement of the
> three QP tasklets by workqueues the only remaining one is the tasklet that
> defers the CQ completion
> handler. This has been in there since the driver went upstream so the
> history of why it is there is lost.
> 
> I notice that the mlx5 driver does have a deferral mechanism for the
> completion handler but the siw driver
> does not. I really do not see what advantage, if any, this has for the rxe
> driver. Perhaps there is some
> reason it shouldn't run in hard interrupt context but the CQ tasklet is a
> soft interrupt so the completion
> handler can't sleep anyway.
> 
> As an experiment I removed the CQ tasklet in the rxe driver and it runs
> fine. In fact the performance is
> slightly better with the completion handler called inline rather than
> deferred to another tasklet.

That is what I would suggest to do. Why would you leave receive
processing or failing send processing w/o creating the CQE and
kicking the CQ handler, if you are in a context with
all information available to build a CQE, signal its availability
to the consumer and kick a user handler if registered and armed?

Only exception I see: If you process the SQ in post_send() user context
and a failure results in immediate CQE creation, direct CQ handler calling
is not allowed - see Documentation/infiniband/core_locking.rst
Not sure though, if rxe allows for direct SQ processing out of user
context.

Cheers,
Bernard.

> If we can eliminate this there won't be anymore tasklets in the rxe driver.
> 
> Does anyone know why the tasklet was put in in the first place?
> 
> Thanks,
> 
> Bob

Re: question about the completion tasklet in the rxe driver

[Bob Pearson]

On 3/16/23 04:40, Bernard Metzler wrote:
> 
> 
>> -----Original Message-----
>> From: Bob Pearson <rpearsonhpe@gmail.com>
>> Sent: Wednesday, 15 March 2023 22:56
>> To: Jason Gunthorpe <jgg@nvidia.com>; Zhu Yanjun <zyjzyj2000@gmail.com>;
>> linux-rdma@vger.kernel.org; Bernard Metzler <BMT@zurich.ibm.com>
>> Subject: [EXTERNAL] question about the completion tasklet in the rxe driver
>>
>> I have a goal of trying to get rid of all the tasklets in the rxe driver
>> and with the replacement of the
>> three QP tasklets by workqueues the only remaining one is the tasklet that
>> defers the CQ completion
>> handler. This has been in there since the driver went upstream so the
>> history of why it is there is lost.
>>
>> I notice that the mlx5 driver does have a deferral mechanism for the
>> completion handler but the siw driver
>> does not. I really do not see what advantage, if any, this has for the rxe
>> driver. Perhaps there is some
>> reason it shouldn't run in hard interrupt context but the CQ tasklet is a
>> soft interrupt so the completion
>> handler can't sleep anyway.
>>
>> As an experiment I removed the CQ tasklet in the rxe driver and it runs
>> fine. In fact the performance is
>> slightly better with the completion handler called inline rather than
>> deferred to another tasklet.
> 
> That is what I would suggest to do. Why would you leave receive
> processing or failing send processing w/o creating the CQE and
> kicking the CQ handler, if you are in a context with
> all information available to build a CQE, signal its availability
> to the consumer and kick a user handler if registered and armed?
> 
> Only exception I see: If you process the SQ in post_send() user context
> and a failure results in immediate CQE creation, direct CQ handler calling
> is not allowed - see Documentation/infiniband/core_locking.rst
> Not sure though, if rxe allows for direct SQ processing out of user
> context.
> 
> Cheers,
> Bernard.

And you did. I am not sure why the mlx5 driver defers the completion handler call
to a tasklet. I could be that it gets called in a hard interrupt and completion
handling is deferred to a soft interrupt context. But for rxe the completion
is always already in a soft interrupt context or a process context.

Bob

> 
>> If we can eliminate this there won't be anymore tasklets in the rxe driver.
>>
>> Does anyone know why the tasklet was put in in the first place?
>>
>> Thanks,
>>
>> Bob

RE: question about the completion tasklet in the rxe driver

[Bernard Metzler]

> -----Original Message-----
> From: Bob Pearson <rpearsonhpe@gmail.com>
> Sent: Thursday, 16 March 2023 18:15
> To: Bernard Metzler <BMT@zurich.ibm.com>; Jason Gunthorpe <jgg@nvidia.com>;
> Zhu Yanjun <zyjzyj2000@gmail.com>; linux-rdma@vger.kernel.org
> Subject: [EXTERNAL] Re: question about the completion tasklet in the rxe
> driver
> 
> On 3/16/23 04:40, Bernard Metzler wrote:
> >
> >
> >> -----Original Message-----
> >> From: Bob Pearson <rpearsonhpe@gmail.com>
> >> Sent: Wednesday, 15 March 2023 22:56
> >> To: Jason Gunthorpe <jgg@nvidia.com>; Zhu Yanjun <zyjzyj2000@gmail.com>;
> >> linux-rdma@vger.kernel.org; Bernard Metzler <BMT@zurich.ibm.com>
> >> Subject: [EXTERNAL] question about the completion tasklet in the rxe
> driver
> >>
> >> I have a goal of trying to get rid of all the tasklets in the rxe driver
> >> and with the replacement of the
> >> three QP tasklets by workqueues the only remaining one is the tasklet
> that
> >> defers the CQ completion
> >> handler. This has been in there since the driver went upstream so the
> >> history of why it is there is lost.
> >>
> >> I notice that the mlx5 driver does have a deferral mechanism for the
> >> completion handler but the siw driver
> >> does not. I really do not see what advantage, if any, this has for the
> rxe
> >> driver. Perhaps there is some
> >> reason it shouldn't run in hard interrupt context but the CQ tasklet is
> a
> >> soft interrupt so the completion
> >> handler can't sleep anyway.
> >>
> >> As an experiment I removed the CQ tasklet in the rxe driver and it runs
> >> fine. In fact the performance is
> >> slightly better with the completion handler called inline rather than
> >> deferred to another tasklet.
> >
> > That is what I would suggest to do. Why would you leave receive
> > processing or failing send processing w/o creating the CQE and
> > kicking the CQ handler, if you are in a context with
> > all information available to build a CQE, signal its availability
> > to the consumer and kick a user handler if registered and armed?
> >
> > Only exception I see: If you process the SQ in post_send() user context
> > and a failure results in immediate CQE creation, direct CQ handler
> calling
> > is not allowed - see Documentation/infiniband/core_locking.rst
> > Not sure though, if rxe allows for direct SQ processing out of user
> > context.
> >
> > Cheers,
> > Bernard.
> 
> And you did. I am not sure why the mlx5 driver defers the completion
> handler call
> to a tasklet. I could be that it gets called in a hard interrupt and
> completion
> handling is deferred to a soft interrupt context. But for rxe the
> completion
> is always already in a soft interrupt context or a process context.
> 
right, go for it ;)


> Bob
> 
> >
> >> If we can eliminate this there won't be anymore tasklets in the rxe
> driver.
> >>
> >> Does anyone know why the tasklet was put in in the first place?
> >>
> >> Thanks,
> >>
> >> Bob

Re: question about the completion tasklet in the rxe driver

[Zhu Yanjun]

在 2023/3/16 5:56, Bob Pearson 写道:
> I have a goal of trying to get rid of all the tasklets in the rxe driver and with the replacement of the

If work queue is used, is it possible to distribute the work queue tasks 
to different cpus evenly in the same numa (if numa is consider)?  ^_^

Zhu Yanjun

> three QP tasklets by workqueues the only remaining one is the tasklet that defers the CQ completion
> handler. This has been in there since the driver went upstream so the history of why it is there is lost.
> 
> I notice that the mlx5 driver does have a deferral mechanism for the completion handler but the siw driver
> does not. I really do not see what advantage, if any, this has for the rxe driver. Perhaps there is some
> reason it shouldn't run in hard interrupt context but the CQ tasklet is a soft interrupt so the completion
> handler can't sleep anyway.
> 
> As an experiment I removed the CQ tasklet in the rxe driver and it runs fine. In fact the performance is
> slightly better with the completion handler called inline rather than deferred to another tasklet.
> If we can eliminate this there won't be anymore tasklets in the rxe driver.
> 
> Does anyone know why the tasklet was put in in the first place?
> 
> Thanks,
> 
> Bob

Re: question about the completion tasklet in the rxe driver

[Jason Gunthorpe]

On Thu, Mar 16, 2023 at 12:14:30PM -0500, Bob Pearson wrote:

> And you did. I am not sure why the mlx5 driver defers the completion handler call
> to a tasklet. I could be that it gets called in a hard interrupt and completion
> handling is deferred to a soft interrupt context. But for rxe the completion
> is always already in a soft interrupt context or a process context.

mlx5_eq_comp_int() is a hard IRQ context

mlx5_cq_tasklet_cb() is in a tasklet context

It has some logic that it only does a certain amount of work per
tasklet call and then it reschedules itself to run again.

mlx5_ib_cq_comp() is in a tasklet context

Jason