Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
* [question]can hard roce and soft roce communicate with each other?
@ 2019-11-22  7:19 wangqi
  2019-11-27  0:53 ` Bart Van Assche
  0 siblings, 1 reply; 7+ messages in thread
From: wangqi @ 2019-11-22  7:19 UTC (permalink / raw)
  To: Leon Romanovsky, linux-rdma

Dear experts on RDMA,

    I'm sorry to bother you again. 

    Do you know how to make soft-roce (on server) can send message 

to the hard-roce (like Mellanox cx4 card) on a client? We tried rdma-core 

25.0 and 26.0. The rdma-core can support both soft-roce and hard-roce.

But it seems that the soft-roce (server) and hard-roce (client) can not 

communicate via "ib_send_bw", "ib_read_bw" and so on, but can 

communicate via "rping". 

    Do you ever try to use soft-roce and hard-roce together? 

Do they work well? I really wonder why they can not communicate with

each other. Best wishes,

Qi


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [question]can hard roce and soft roce communicate with each other?
  2019-11-22  7:19 [question]can hard roce and soft roce communicate with each other? wangqi
@ 2019-11-27  0:53 ` Bart Van Assche
  2019-11-27 11:10   ` Leon Romanovsky
  0 siblings, 1 reply; 7+ messages in thread
From: Bart Van Assche @ 2019-11-27  0:53 UTC (permalink / raw)
  To: wangqi, Leon Romanovsky, linux-rdma

On 11/21/19 11:19 PM, wangqi wrote:
>      Do you know how to make soft-roce (on server) can send message
> to the hard-roce (like Mellanox cx4 card) on a client? We tried rdma-core
> 25.0 and 26.0. The rdma-core can support both soft-roce and hard-roce.
> 
> But it seems that the soft-roce (server) and hard-roce (client) can not
> communicate via "ib_send_bw", "ib_read_bw" and so on, but can
> communicate via "rping".
> 
>      Do you ever try to use soft-roce and hard-roce together?
> Do they work well? I really wonder why they can not communicate with
> each other. Best wishes,

I think this should be possible. The diagram on the following web page 
shows a RoCE NIC and softROCE connected to each other:

http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience-rdma/

Bart.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [question]can hard roce and soft roce communicate with each other?
  2019-11-27  0:53 ` Bart Van Assche
@ 2019-11-27 11:10   ` Leon Romanovsky
  2019-11-27 14:24     ` Steve Wise
  0 siblings, 1 reply; 7+ messages in thread
From: Leon Romanovsky @ 2019-11-27 11:10 UTC (permalink / raw)
  To: Bart Van Assche; +Cc: wangqi, linux-rdma

On Tue, Nov 26, 2019 at 04:53:14PM -0800, Bart Van Assche wrote:
> On 11/21/19 11:19 PM, wangqi wrote:
> >      Do you know how to make soft-roce (on server) can send message
> > to the hard-roce (like Mellanox cx4 card) on a client? We tried rdma-core
> > 25.0 and 26.0. The rdma-core can support both soft-roce and hard-roce.
> >
> > But it seems that the soft-roce (server) and hard-roce (client) can not
> > communicate via "ib_send_bw", "ib_read_bw" and so on, but can
> > communicate via "rping".
> >
> >      Do you ever try to use soft-roce and hard-roce together?
> > Do they work well? I really wonder why they can not communicate with
> > each other. Best wishes,
>
> I think this should be possible. The diagram on the following web page shows
> a RoCE NIC and softROCE connected to each other:
>
> http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience-rdma/

It should work, but it didn't work for me now :)

Thanks

>
> Bart.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [question]can hard roce and soft roce communicate with each other?
  2019-11-27 11:10   ` Leon Romanovsky
@ 2019-11-27 14:24     ` Steve Wise
  2019-12-03  0:56       ` Bart Van Assche
  0 siblings, 1 reply; 7+ messages in thread
From: Steve Wise @ 2019-11-27 14:24 UTC (permalink / raw)
  To: Leon Romanovsky; +Cc: Bart Van Assche, wangqi, linux-rdma

I've recently uncovered a bug in RXE that causes iCRC errors when
running between RXE and a correct RoCE implementation.  The bug is
that RXE is not including pad bytes in its iCRC calculations.  So if
the application payload is not 4B aligned then you'll hit this bug.
You can see this by running ib_write_bw, for example, between mlnx_ib
and rxe.

works:  ib_write_bw -s 32 -n 5
fails: ib_write_bw -s 33 -n 5

I'll post a patch this coming weekend hopefully.

Steve.

On Wed, Nov 27, 2019 at 5:10 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Nov 26, 2019 at 04:53:14PM -0800, Bart Van Assche wrote:
> > On 11/21/19 11:19 PM, wangqi wrote:
> > >      Do you know how to make soft-roce (on server) can send message
> > > to the hard-roce (like Mellanox cx4 card) on a client? We tried rdma-core
> > > 25.0 and 26.0. The rdma-core can support both soft-roce and hard-roce.
> > >
> > > But it seems that the soft-roce (server) and hard-roce (client) can not
> > > communicate via "ib_send_bw", "ib_read_bw" and so on, but can
> > > communicate via "rping".
> > >
> > >      Do you ever try to use soft-roce and hard-roce together?
> > > Do they work well? I really wonder why they can not communicate with
> > > each other. Best wishes,
> >
> > I think this should be possible. The diagram on the following web page shows
> > a RoCE NIC and softROCE connected to each other:
> >
> > http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience-rdma/
>
> It should work, but it didn't work for me now :)
>
> Thanks
>
> >
> > Bart.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [question]can hard roce and soft roce communicate with each other?
  2019-11-27 14:24     ` Steve Wise
@ 2019-12-03  0:56       ` Bart Van Assche
  2019-12-03  1:57         ` Steve Wise
  0 siblings, 1 reply; 7+ messages in thread
From: Bart Van Assche @ 2019-12-03  0:56 UTC (permalink / raw)
  To: Steve Wise, Leon Romanovsky; +Cc: wangqi, linux-rdma

On 11/27/19 6:24 AM, Steve Wise wrote:
> I've recently uncovered a bug in RXE that causes iCRC errors when
> running between RXE and a correct RoCE implementation.  The bug is
> that RXE is not including pad bytes in its iCRC calculations.  So if
> the application payload is not 4B aligned then you'll hit this bug.
> You can see this by running ib_write_bw, for example, between mlnx_ib
> and rxe.
> 
> works:  ib_write_bw -s 32 -n 5
> fails: ib_write_bw -s 33 -n 5
> 
> I'll post a patch this coming weekend hopefully.
  Hi Steve,

Will that patch support coexistence of softRoCE implementations that use 
different CRC calculation methods?

Thanks,

Bart.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [question]can hard roce and soft roce communicate with each other?
  2019-12-03  0:56       ` Bart Van Assche
@ 2019-12-03  1:57         ` Steve Wise
  2019-12-03 21:48           ` Steve Wise
  0 siblings, 1 reply; 7+ messages in thread
From: Steve Wise @ 2019-12-03  1:57 UTC (permalink / raw)
  To: Bart Van Assche; +Cc: Leon Romanovsky, wangqi, linux-rdma

Hey Bart,

Well, as long as every implementation supports proper iCRC and RoCE
standards, I don't see how there could be a problem...

Let me send my patch out now...


On Mon, Dec 2, 2019 at 6:57 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 11/27/19 6:24 AM, Steve Wise wrote:
> > I've recently uncovered a bug in RXE that causes iCRC errors when
> > running between RXE and a correct RoCE implementation.  The bug is
> > that RXE is not including pad bytes in its iCRC calculations.  So if
> > the application payload is not 4B aligned then you'll hit this bug.
> > You can see this by running ib_write_bw, for example, between mlnx_ib
> > and rxe.
> >
> > works:  ib_write_bw -s 32 -n 5
> > fails: ib_write_bw -s 33 -n 5
> >
> > I'll post a patch this coming weekend hopefully.
>   Hi Steve,
>
> Will that patch support coexistence of softRoCE implementations that use
> different CRC calculation methods?
>
> Thanks,
>
> Bart.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [question]can hard roce and soft roce communicate with each other?
  2019-12-03  1:57         ` Steve Wise
@ 2019-12-03 21:48           ` Steve Wise
  0 siblings, 0 replies; 7+ messages in thread
From: Steve Wise @ 2019-12-03 21:48 UTC (permalink / raw)
  To: Bart Van Assche; +Cc: Leon Romanovsky, wangqi, linux-rdma

On Mon, Dec 2, 2019 at 7:57 PM Steve Wise <larrystevenwise@gmail.com> wrote:
>
> Hey Bart,
>
> Well, as long as every implementation supports proper iCRC and RoCE
> standards, I don't see how there could be a problem...
>
> Let me send my patch out now...
>
>
> On Mon, Dec 2, 2019 at 6:57 PM Bart Van Assche <bvanassche@acm.org> wrote:
> >
> > On 11/27/19 6:24 AM, Steve Wise wrote:
> > > I've recently uncovered a bug in RXE that causes iCRC errors when
> > > running between RXE and a correct RoCE implementation.  The bug is
> > > that RXE is not including pad bytes in its iCRC calculations.  So if
> > > the application payload is not 4B aligned then you'll hit this bug.
> > > You can see this by running ib_write_bw, for example, between mlnx_ib
> > > and rxe.
> > >
> > > works:  ib_write_bw -s 32 -n 5
> > > fails: ib_write_bw -s 33 -n 5
> > >
> > > I'll post a patch this coming weekend hopefully.
> >   Hi Steve,
> >
> > Will that patch support coexistence of softRoCE implementations that use
> > different CRC calculation methods?
> >
> > Thanks,
> >
> > Bart.

In case anyone missed it, I posted my fix for unaligned payloads:

https://www.spinics.net/lists/linux-rdma/msg86758.html

Steve.

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, back to index

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-22  7:19 [question]can hard roce and soft roce communicate with each other? wangqi
2019-11-27  0:53 ` Bart Van Assche
2019-11-27 11:10   ` Leon Romanovsky
2019-11-27 14:24     ` Steve Wise
2019-12-03  0:56       ` Bart Van Assche
2019-12-03  1:57         ` Steve Wise
2019-12-03 21:48           ` Steve Wise

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git