[question]can hard roce and soft roce communicate with each other?

[question]can hard roce and soft roce communicate with each other?

[wangqi]

Dear experts on RDMA，

    I'm sorry to bother you again. 

    Do you know how to make soft-roce (on server) can send message 

to the hard-roce (like Mellanox cx4 card) on a client? We tried rdma-core 

25.0 and 26.0. The rdma-core can support both soft-roce and hard-roce.

But it seems that the soft-roce (server) and hard-roce (client) can not 

communicate via "ib_send_bw", "ib_read_bw" and so on, but can 

communicate via "rping". 

    Do you ever try to use soft-roce and hard-roce together? 

Do they work well? I really wonder why they can not communicate with

each other. Best wishes,

Qi

Re: [question]can hard roce and soft roce communicate with each other?

[Bart Van Assche]

On 11/21/19 11:19 PM, wangqi wrote:
>      Do you know how to make soft-roce (on server) can send message
> to the hard-roce (like Mellanox cx4 card) on a client? We tried rdma-core
> 25.0 and 26.0. The rdma-core can support both soft-roce and hard-roce.
> 
> But it seems that the soft-roce (server) and hard-roce (client) can not
> communicate via "ib_send_bw", "ib_read_bw" and so on, but can
> communicate via "rping".
> 
>      Do you ever try to use soft-roce and hard-roce together?
> Do they work well? I really wonder why they can not communicate with
> each other. Best wishes,

I think this should be possible. The diagram on the following web page 
shows a RoCE NIC and softROCE connected to each other:

http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience-rdma/

Bart.

Re: [question]can hard roce and soft roce communicate with each other?

[Leon Romanovsky]

On Tue, Nov 26, 2019 at 04:53:14PM -0800, Bart Van Assche wrote:
> On 11/21/19 11:19 PM, wangqi wrote:
> >      Do you know how to make soft-roce (on server) can send message
> > to the hard-roce (like Mellanox cx4 card) on a client? We tried rdma-core
> > 25.0 and 26.0. The rdma-core can support both soft-roce and hard-roce.
> >
> > But it seems that the soft-roce (server) and hard-roce (client) can not
> > communicate via "ib_send_bw", "ib_read_bw" and so on, but can
> > communicate via "rping".
> >
> >      Do you ever try to use soft-roce and hard-roce together?
> > Do they work well? I really wonder why they can not communicate with
> > each other. Best wishes,
>
> I think this should be possible. The diagram on the following web page shows
> a RoCE NIC and softROCE connected to each other:
>
> http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience-rdma/

It should work, but it didn't work for me now :)

Thanks

>
> Bart.

Re: [question]can hard roce and soft roce communicate with each other?

[Steve Wise]

I've recently uncovered a bug in RXE that causes iCRC errors when
running between RXE and a correct RoCE implementation.  The bug is
that RXE is not including pad bytes in its iCRC calculations.  So if
the application payload is not 4B aligned then you'll hit this bug.
You can see this by running ib_write_bw, for example, between mlnx_ib
and rxe.

works:  ib_write_bw -s 32 -n 5
fails: ib_write_bw -s 33 -n 5

I'll post a patch this coming weekend hopefully.

Steve.

On Wed, Nov 27, 2019 at 5:10 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Nov 26, 2019 at 04:53:14PM -0800, Bart Van Assche wrote:
> > On 11/21/19 11:19 PM, wangqi wrote:
> > >      Do you know how to make soft-roce (on server) can send message
> > > to the hard-roce (like Mellanox cx4 card) on a client? We tried rdma-core
> > > 25.0 and 26.0. The rdma-core can support both soft-roce and hard-roce.
> > >
> > > But it seems that the soft-roce (server) and hard-roce (client) can not
> > > communicate via "ib_send_bw", "ib_read_bw" and so on, but can
> > > communicate via "rping".
> > >
> > >      Do you ever try to use soft-roce and hard-roce together?
> > > Do they work well? I really wonder why they can not communicate with
> > > each other. Best wishes,
> >
> > I think this should be possible. The diagram on the following web page shows
> > a RoCE NIC and softROCE connected to each other:
> >
> > http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience-rdma/
>
> It should work, but it didn't work for me now :)
>
> Thanks
>
> >
> > Bart.

Re: [question]can hard roce and soft roce communicate with each other?

[Bart Van Assche]

On 11/27/19 6:24 AM, Steve Wise wrote:
> I've recently uncovered a bug in RXE that causes iCRC errors when
> running between RXE and a correct RoCE implementation.  The bug is
> that RXE is not including pad bytes in its iCRC calculations.  So if
> the application payload is not 4B aligned then you'll hit this bug.
> You can see this by running ib_write_bw, for example, between mlnx_ib
> and rxe.
> 
> works:  ib_write_bw -s 32 -n 5
> fails: ib_write_bw -s 33 -n 5
> 
> I'll post a patch this coming weekend hopefully.
  Hi Steve,

Will that patch support coexistence of softRoCE implementations that use 
different CRC calculation methods?

Thanks,

Bart.

Re: [question]can hard roce and soft roce communicate with each other?

[Steve Wise]

Hey Bart,

Well, as long as every implementation supports proper iCRC and RoCE
standards, I don't see how there could be a problem...

Let me send my patch out now...


On Mon, Dec 2, 2019 at 6:57 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 11/27/19 6:24 AM, Steve Wise wrote:
> > I've recently uncovered a bug in RXE that causes iCRC errors when
> > running between RXE and a correct RoCE implementation.  The bug is
> > that RXE is not including pad bytes in its iCRC calculations.  So if
> > the application payload is not 4B aligned then you'll hit this bug.
> > You can see this by running ib_write_bw, for example, between mlnx_ib
> > and rxe.
> >
> > works:  ib_write_bw -s 32 -n 5
> > fails: ib_write_bw -s 33 -n 5
> >
> > I'll post a patch this coming weekend hopefully.
>   Hi Steve,
>
> Will that patch support coexistence of softRoCE implementations that use
> different CRC calculation methods?
>
> Thanks,
>
> Bart.

Re: [question]can hard roce and soft roce communicate with each other?

[Steve Wise]

On Mon, Dec 2, 2019 at 7:57 PM Steve Wise <larrystevenwise@gmail.com> wrote:
>
> Hey Bart,
>
> Well, as long as every implementation supports proper iCRC and RoCE
> standards, I don't see how there could be a problem...
>
> Let me send my patch out now...
>
>
> On Mon, Dec 2, 2019 at 6:57 PM Bart Van Assche <bvanassche@acm.org> wrote:
> >
> > On 11/27/19 6:24 AM, Steve Wise wrote:
> > > I've recently uncovered a bug in RXE that causes iCRC errors when
> > > running between RXE and a correct RoCE implementation.  The bug is
> > > that RXE is not including pad bytes in its iCRC calculations.  So if
> > > the application payload is not 4B aligned then you'll hit this bug.
> > > You can see this by running ib_write_bw, for example, between mlnx_ib
> > > and rxe.
> > >
> > > works:  ib_write_bw -s 32 -n 5
> > > fails: ib_write_bw -s 33 -n 5
> > >
> > > I'll post a patch this coming weekend hopefully.
> >   Hi Steve,
> >
> > Will that patch support coexistence of softRoCE implementations that use
> > different CRC calculation methods?
> >
> > Thanks,
> >
> > Bart.

In case anyone missed it, I posted my fix for unaligned payloads:

https://www.spinics.net/lists/linux-rdma/msg86758.html

Steve.