software iwarp stack update

software iwarp stack update

[Bernard Metzler]

Earlier this year, we announced the availability of an open source,
full software implementation of the iWARP RDMA protocol stack - see
my email "software iwarp stack" from March 14th at the linux-rdma list
(http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg02940.html)
While since then working on performance and stability, we provided
some source code updates. Current user and kernel code is available at
gitorious.org/softiwarp. Please see the CHANGES file in the
kernel/ directory for a summary of the most recent changes.

For more convenient testing, the latest update now allows for a
stand-alone build of the kernel module without full kernel source
code access. We tested the code with kernel version 2.6.34. If
you are interested in a full software RDMA stack on Ethernet,
please try it out.

In the hope of providing useful information, I put
netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org on copy. Subscribers of this list,
please put me on private cc in case you reply or comment, since
I am not subscribed to the list.
We would be more than happy if you netdev folks would consider
a hardware independent RDMA kernel service as something useful and
potentially to be integrated into the mainline network stack.

Why might it be useful?
A software RDMA stack makes the semantic advantages of
asynchronous and one-sided communication available while obsoleting
the need to deploy dedicated RDMA hardware or any protocol offloading
(while not matching the lowest delay numbers of real RDMA hardware).
Implementing the IETF's iWARP protocol stack on top of TCP kernel
sockets, softiwarp integrates with the open fabrics environment
and thus exports the RDMA kernel and user verbs interface.

The efficiency of the Linux TCP/IP network stack together with intrinsic
advantages of the RDMA communication model (async. posting of work
and reaping of work completions, transfer of send buffer ownership
to the kernel which enables zero copy transmit, peer data placement
without application scheduling, one-sided remote read operations etc.)
can result in improved application-to-application performance and
less CPU load, while using the unchanged kernel TCP stack.

A software RDMA stack might promote wider RDMA deployment,
since when using the host TCP stack, it enables RDMA semantic
independent of dedicated hardware. softiwarp peers with real
RNICs (tested with Chelsio's T3 adapter).

softiwarp is still work in progress and we are very thankful for any
suggestions/comments/bug reports. Please advise how we should proceed
to bring the stack further to your attention. Would it be useful to
provide patches against the current stable kernel version or the next
release candidate?

Many thanks,
Bernard.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Nicholas A. Bellinger]

On Wed, 2010-09-22 at 10:19 +0200, Bernard Metzler wrote:
> Earlier this year, we announced the availability of an open source,
> full software implementation of the iWARP RDMA protocol stack - see
> my email "software iwarp stack" from March 14th at the linux-rdma list
> (http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg02940.html)
> While since then working on performance and stability, we provided
> some source code updates. Current user and kernel code is available at
> gitorious.org/softiwarp. Please see the CHANGES file in the
> kernel/ directory for a summary of the most recent changes.
> 
> For more convenient testing, the latest update now allows for a
> stand-alone build of the kernel module without full kernel source
> code access. We tested the code with kernel version 2.6.34. If
> you are interested in a full software RDMA stack on Ethernet,
> please try it out.
> 
> In the hope of providing useful information, I put
> netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org on copy. Subscribers of this list,
> please put me on private cc in case you reply or comment, since
> I am not subscribed to the list.
> We would be more than happy if you netdev folks would consider
> a hardware independent RDMA kernel service as something useful and
> potentially to be integrated into the mainline network stack.
> 
> Why might it be useful?
> A software RDMA stack makes the semantic advantages of
> asynchronous and one-sided communication available while obsoleting
> the need to deploy dedicated RDMA hardware or any protocol offloading
> (while not matching the lowest delay numbers of real RDMA hardware).
> Implementing the IETF's iWARP protocol stack on top of TCP kernel
> sockets, softiwarp integrates with the open fabrics environment
> and thus exports the RDMA kernel and user verbs interface.
> 
> The efficiency of the Linux TCP/IP network stack together with intrinsic
> advantages of the RDMA communication model (async. posting of work
> and reaping of work completions, transfer of send buffer ownership
> to the kernel which enables zero copy transmit, peer data placement
> without application scheduling, one-sided remote read operations etc.)
> can result in improved application-to-application performance and
> less CPU load, while using the unchanged kernel TCP stack.
> 
> A software RDMA stack might promote wider RDMA deployment,
> since when using the host TCP stack, it enables RDMA semantic
> independent of dedicated hardware. softiwarp peers with real
> RNICs (tested with Chelsio's T3 adapter).
> 
> softiwarp is still work in progress and we are very thankful for any
> suggestions/comments/bug reports. Please advise how we should proceed
> to bring the stack further to your attention. Would it be useful to
> provide patches against the current stable kernel version or the next
> release candidate?
> 

Hi Bernard,

So what I would recommend doing here to make things more appealing to
DaveM and other interested NetDev folks would be to clone a seperate
tree from the net-2.6.git or net-next-2.6.git repositories and include
the softiwarp/kernel.git code into a fresh 'in-kernel' clone tracking
the latest netdev code, and then keep git rebase'ing against DaveM's
last changes and update your local tree to the lastest netdev code.

Of course you will want to remove all of the 'out of tree' LINUX_VERSION
build macros and any other legacy bits to follow mainline kernel
convention for your 'in-kernel' softiwarp tree.

Best,

--nab


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Steve Wise]

On 09/22/2010 03:35 PM, Nicholas A. Bellinger wrote:
> On Wed, 2010-09-22 at 10:19 +0200, Bernard Metzler wrote:
>    
>> Earlier this year, we announced the availability of an open source,
>> full software implementation of the iWARP RDMA protocol stack - see
>> my email "software iwarp stack" from March 14th at the linux-rdma list
>> (http://www.mail-archive.com/linux-rdma@vger.kernel.org/msg02940.html)
>> While since then working on performance and stability, we provided
>> some source code updates. Current user and kernel code is available at
>> gitorious.org/softiwarp. Please see the CHANGES file in the
>> kernel/ directory for a summary of the most recent changes.
>>
>> For more convenient testing, the latest update now allows for a
>> stand-alone build of the kernel module without full kernel source
>> code access. We tested the code with kernel version 2.6.34. If
>> you are interested in a full software RDMA stack on Ethernet,
>> please try it out.
>>
>> In the hope of providing useful information, I put
>> netdev@vger.kernel.org on copy. Subscribers of this list,
>> please put me on private cc in case you reply or comment, since
>> I am not subscribed to the list.
>> We would be more than happy if you netdev folks would consider
>> a hardware independent RDMA kernel service as something useful and
>> potentially to be integrated into the mainline network stack.
>>
>> Why might it be useful?
>> A software RDMA stack makes the semantic advantages of
>> asynchronous and one-sided communication available while obsoleting
>> the need to deploy dedicated RDMA hardware or any protocol offloading
>> (while not matching the lowest delay numbers of real RDMA hardware).
>> Implementing the IETF's iWARP protocol stack on top of TCP kernel
>> sockets, softiwarp integrates with the open fabrics environment
>> and thus exports the RDMA kernel and user verbs interface.
>>
>> The efficiency of the Linux TCP/IP network stack together with intrinsic
>> advantages of the RDMA communication model (async. posting of work
>> and reaping of work completions, transfer of send buffer ownership
>> to the kernel which enables zero copy transmit, peer data placement
>> without application scheduling, one-sided remote read operations etc.)
>> can result in improved application-to-application performance and
>> less CPU load, while using the unchanged kernel TCP stack.
>>
>> A software RDMA stack might promote wider RDMA deployment,
>> since when using the host TCP stack, it enables RDMA semantic
>> independent of dedicated hardware. softiwarp peers with real
>> RNICs (tested with Chelsio's T3 adapter).
>>
>> softiwarp is still work in progress and we are very thankful for any
>> suggestions/comments/bug reports. Please advise how we should proceed
>> to bring the stack further to your attention. Would it be useful to
>> provide patches against the current stable kernel version or the next
>> release candidate?
>>
>>      
> Hi Bernard,
>
> So what I would recommend doing here to make things more appealing to
> DaveM and other interested NetDev folks would be to clone a seperate
> tree from the net-2.6.git or net-next-2.6.git repositories and include
> the softiwarp/kernel.git code into a fresh 'in-kernel' clone tracking
> the latest netdev code, and then keep git rebase'ing against DaveM's
> last changes and update your local tree to the lastest netdev code.
>
> Of course you will want to remove all of the 'out of tree' LINUX_VERSION
> build macros and any other legacy bits to follow mainline kernel
> convention for your 'in-kernel' softiwarp tree.
>
>    

And then post a patch series for review.


Steve.

Re: software iwarp stack update

[Bernard Metzler]

linux-rdma-owner@vger.kernel.org wrote on 09/22/2010 10:42:18 PM:

> On 09/22/2010 03:35 PM, Nicholas A. Bellinger wrote:
> > On Wed, 2010-09-22 at 10:19 +0200, Bernard Metzler wrote:
> >
> >> Earlier this year, we announced the availability of an open source,
> >> full software implementation of the iWARP RDMA protocol stack - see
> >> my email "software iwarp stack" from March 14th at the linux-rdma list
> >> (http://www.mail-archive.com/linux-rdma@vger.kernel.org/msg02940.html)
> >> While since then working on performance and stability, we provided
> >> some source code updates. Current user and kernel code is available at
> >> gitorious.org/softiwarp. Please see the CHANGES file in the
> >> kernel/ directory for a summary of the most recent changes.
> >>
> >> For more convenient testing, the latest update now allows for a
> >> stand-alone build of the kernel module without full kernel source
> >> code access. We tested the code with kernel version 2.6.34. If
> >> you are interested in a full software RDMA stack on Ethernet,
> >> please try it out.
> >>
> >> In the hope of providing useful information, I put
> >> netdev@vger.kernel.org on copy. Subscribers of this list,
> >> please put me on private cc in case you reply or comment, since
> >> I am not subscribed to the list.
> >> We would be more than happy if you netdev folks would consider
> >> a hardware independent RDMA kernel service as something useful and
> >> potentially to be integrated into the mainline network stack.
> >>
> >> Why might it be useful?
> >> A software RDMA stack makes the semantic advantages of
> >> asynchronous and one-sided communication available while obsoleting
> >> the need to deploy dedicated RDMA hardware or any protocol offloading
> >> (while not matching the lowest delay numbers of real RDMA hardware).
> >> Implementing the IETF's iWARP protocol stack on top of TCP kernel
> >> sockets, softiwarp integrates with the open fabrics environment
> >> and thus exports the RDMA kernel and user verbs interface.
> >>
> >> The efficiency of the Linux TCP/IP network stack together with
intrinsic
> >> advantages of the RDMA communication model (async. posting of work
> >> and reaping of work completions, transfer of send buffer ownership
> >> to the kernel which enables zero copy transmit, peer data placement
> >> without application scheduling, one-sided remote read operations etc.)
> >> can result in improved application-to-application performance and
> >> less CPU load, while using the unchanged kernel TCP stack.
> >>
> >> A software RDMA stack might promote wider RDMA deployment,
> >> since when using the host TCP stack, it enables RDMA semantic
> >> independent of dedicated hardware. softiwarp peers with real
> >> RNICs (tested with Chelsio's T3 adapter).
> >>
> >> softiwarp is still work in progress and we are very thankful for any
> >> suggestions/comments/bug reports. Please advise how we should proceed
> >> to bring the stack further to your attention. Would it be useful to
> >> provide patches against the current stable kernel version or the next
> >> release candidate?
> >>
> >>
> > Hi Bernard,
> >
> > So what I would recommend doing here to make things more appealing to
> > DaveM and other interested NetDev folks would be to clone a seperate
> > tree from the net-2.6.git or net-next-2.6.git repositories and include
> > the softiwarp/kernel.git code into a fresh 'in-kernel' clone tracking
> > the latest netdev code, and then keep git rebase'ing against DaveM's
> > last changes and update your local tree to the lastest netdev code.
> >
> > Of course you will want to remove all of the 'out of tree'
LINUX_VERSION
> > build macros and any other legacy bits to follow mainline kernel
> > convention for your 'in-kernel' softiwarp tree.
> >
> >
>
> And then post a patch series for review.
>

All,

Yes, ok, thats what I will do now.
Many thanks for the helpful and encouraging replies.

Bernard.

Software iWARP (SIW) patches

[Bernard Metzler]

All,
many thanks for your suggestions and help in preparing a set of
patches to add an in-kernel, full-software iWARP stack. As suggested,
the patches are against the current net-next-2.6.git repository,
all version dependencies etc. are removed. To avoid redundancy,
the patches will be posted to the netdev list only. Please keep
me on cc on any replies, since I am not subscribed to netdev.

We post the patches in the hope to make a useful contribution
to the Linux networking stack, and we hope to see SoftiWARP
becoming part of a future mainline Linux. Of course we are very open
to all discussion, suggestions and comments to improve quality and
completeness of the current code. Many thanks in advance!

We will keep www.gitorious.org/softiwarp updated to make a version
of SoftiWARP kernel and user code available for immediate testing
on a running system.
The code at gitorious differs from the current patch in its extensions
to support older kernel versions. All these extensions are #ifdef'ed
and are not part of the upcoming patch series.
www.gitorious.org/softiwarp/userlib remains the current source of the
user library for SoftiWARP.

Please find below both git shortlog and diffstat for overview.


Many Thanks,
Bernard.
--

Bernard Metzler (13):
      SIW: Kconfig and Makefile
      SIW: iWARP Protocol headers
      SIW: Main header file
      SIW: Module initialization
      SIW: User interface
      SIW: Connection management
      SIW: Object management
      SIW: Queue pair
      SIW: Completion queue
      SIW: Transmit path
      SIW: Receive path
      SIW: Debugging and Tracing
      SIW: Documentation (initial)

 Documentation/networking/siw.txt      |   91 ++
 drivers/infiniband/Kconfig            |    1 +
 drivers/infiniband/Makefile           |    1 +
 drivers/infiniband/hw/siw/Kconfig     |   14 +
 drivers/infiniband/hw/siw/Makefile    |    5 +
 drivers/infiniband/hw/siw/iwarp.h     |  324 ++++++
 drivers/infiniband/hw/siw/siw.h       |  816 ++++++++++++++
 drivers/infiniband/hw/siw/siw_ae.c    |   96 ++
 drivers/infiniband/hw/siw/siw_cm.c    | 1939 +++++++++++++++++++++++++++++
++++
 drivers/infiniband/hw/siw/siw_cm.h    |  155 +++
 drivers/infiniband/hw/siw/siw_cq.c    |  243 ++++
 drivers/infiniband/hw/siw/siw_debug.c |  198 ++++
 drivers/infiniband/hw/siw/siw_debug.h |  159 +++
 drivers/infiniband/hw/siw/siw_main.c  |  440 ++++++++
 drivers/infiniband/hw/siw/siw_obj.c   |  499 +++++++++
 drivers/infiniband/hw/siw/siw_obj.h   |  109 ++
 drivers/infiniband/hw/siw/siw_qp.c    |  989 +++++++++++++++++
 drivers/infiniband/hw/siw/siw_qp_rx.c | 1493 +++++++++++++++++++++++++
 drivers/infiniband/hw/siw/siw_qp_tx.c | 1309 ++++++++++++++++++++++
 drivers/infiniband/hw/siw/siw_user.h  |   66 ++
 drivers/infiniband/hw/siw/siw_verbs.c | 1564 ++++++++++++++++++++++++++
 drivers/infiniband/hw/siw/siw_verbs.h |   96 ++
 22 files changed, 10607 insertions(+), 0 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Software iWARP (SIW) patches

[Hefty, Sean]

> To avoid redundancy, the patches will be posted to the netdev list only.

Please copy Linux-rdma on the patches.

Re: software iwarp stack update

[Bart Van Assche]

On Wed, Sep 22, 2010 at 10:19 AM, Bernard Metzler <BMT-OA+xvbQnYDHMbYB6QlFGEg@public.gmane.org> wrote:
> Earlier this year, we announced the availability of an open source,
> full software implementation of the iWARP RDMA protocol stack - see
> my email "software iwarp stack" from March 14th at the linux-rdma list
> (http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg02940.html)
> While since then working on performance and stability, we provided
> some source code updates. Current user and kernel code is available at
> gitorious.org/softiwarp. Please see the CHANGES file in the
> kernel/ directory for a summary of the most recent changes.

(replying to an e-mail of about nine months ago)

Hello Bernard,

It has been pretty quiet recently around the "siw" driver. Have you
had the chance to make any progress recently on this driver ?

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bernard Metzler]

Bart,
many thanks for polling. About nine months ago we sent an initial
'softiwarp' patch to netdev and linux-rdma. We got encouraging
feedback including very helpful advice on how to improve that code.
I think we fixed all issues which were detected by the community
but somehow missed the point to feed back the code. Sorry about that.
Due to pressure from other projects softiwarp went a little
down on my priority list. But its immediately back when you are asking ;)

What we currently have is a stack which - due to requirements
from other projects - evolved further to now include also 
some kernel client support (tested with NFSv4). We also fixed
issues in the connection management code path, which improved
stability. We also added some siw object statistics for debugging.

As mentioned, while keeping gitorious.org/softiwarp up to date
(it includes the latest changes) we missed to come back to the
list. I will now resume working on that, dedicating some more
of my time to it. Let me rebase the code with the current kernel
and send a new patch list later next week. 

Many thanks, and sorry for the long-lasting silence,
Bernard.

linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org wrote on 06/04/2011 08:01:45 PM:

> Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org> 
> Sent by: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> 
> 06/04/2011 08:02 PM
> 
> To
> 
> Bernard Metzler <BMT-OA+xvbQnYDHMbYB6QlFGEg@public.gmane.org>
> 
> cc
> 
> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> 
> Subject
> 
> Re: software iwarp stack update
> 
> On Wed, Sep 22, 2010 at 10:19 AM, Bernard Metzler <BMT-OA+xvbQnYDHMbYB6QlFGEg@public.gmane.org> 
wrote:
> > Earlier this year, we announced the availability of an open source,
> > full software implementation of the iWARP RDMA protocol stack - see
> > my email "software iwarp stack" from March 14th at the linux-rdma list
> > (http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg02940.html)
> > While since then working on performance and stability, we provided
> > some source code updates. Current user and kernel code is available at
> > gitorious.org/softiwarp. Please see the CHANGES file in the
> > kernel/ directory for a summary of the most recent changes.
> 
> (replying to an e-mail of about nine months ago)
> 
> Hello Bernard,
> 
> It has been pretty quiet recently around the "siw" driver. Have you
> had the chance to make any progress recently on this driver ?
> 
> Thanks,
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bernard Metzler]

Bart, all,
after rebasing to davem/net-next-2.6.git, I formatted a patch adding
to the initial 'siw' driver posting from last October. Given the rather
substantial amount of changes and time elapsed, I am not sure if people
would better like a complete re-posting of all files.
If I don't hear objection before tomorrow I would post it as an
increment - as summarized below.

Many thanks for advice,
Bernard.

 drivers/infiniband/hw/siw/Makefile |    2 +-
 drivers/infiniband/hw/siw/iwarp.h |  100 +++++++++++++++++
+------------------
 drivers/infiniband/hw/siw/siw.h |  128 +++++++++++++++++
+--------------------
 drivers/infiniband/hw/siw/siw_main.c |  229 +++++++++++++++++++++++++++
+------
 drivers/infiniband/hw/siw/siw_ae.c    |    2 +-
 drivers/infiniband/hw/siw/siw_verbs.c |  572 ++++++++++++++++++++
+------------
 drivers/infiniband/hw/siw/siw_verbs.h |    3 +
 drivers/infiniband/hw/siw/siw_cm.c | 1158 +++++++++++++++++
+------------------
 drivers/infiniband/hw/siw/siw_cm.h |   38 +-
 drivers/infiniband/hw/siw/siw_obj.c |  165 +++++++++++++
+---------------------
 drivers/infiniband/hw/siw/siw_obj.h |   34 +++++++-
 drivers/infiniband/hw/siw/siw_qp.c |  128 ++++++++++++++++++++
+---------------
 drivers/infiniband/hw/siw/siw_cq.c |   24 ++++--------------------
 drivers/infiniband/hw/siw/siw_qp_tx.c |  147 ++++++++++++++++++
+--------------
 drivers/infiniband/hw/siw/siw_qp_rx.c |  234 ++++++++++++++++++++
+------------
 Documentation/networking/siw.txt |   62 +++++++++++++++++++++++++
+-----------
 drivers/infiniband/hw/siw/siw_mem.c |  178 +++++++++++++++++++++++++++++++
++++


linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org wrote on 06/04/2011 08:01:45 PM:

> On Wed, Sep 22, 2010 at 10:19 AM, Bernard Metzler <BMT-OA+xvbQnYDHMbYB6QlFGEg@public.gmane.org>
wrote:
> > Earlier this year, we announced the availability of an open source,
> > full software implementation of the iWARP RDMA protocol stack - see
> > my email "software iwarp stack" from March 14th at the linux-rdma list
> > (http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg02940.html)
> > While since then working on performance and stability, we provided
> > some source code updates. Current user and kernel code is available at
> > gitorious.org/softiwarp. Please see the CHANGES file in the
> > kernel/ directory for a summary of the most recent changes.
>
> (replying to an e-mail of about nine months ago)
>
> Hello Bernard,
>
> It has been pretty quiet recently around the "siw" driver. Have you
> had the chance to make any progress recently on this driver ?
>
> Thanks,
>
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bart Van Assche]

On Wed, Jun 15, 2011 at 5:43 PM, Bernard Metzler <BMT@zurich.ibm.com> wrote:
> after rebasing to davem/net-next-2.6.git, I formatted a patch adding
> to the initial 'siw' driver posting from last October. Given the rather
> substantial amount of changes and time elapsed, I am not sure if people
> would better like a complete re-posting of all files.
> If I don't hear objection before tomorrow I would post it as an
> increment - as summarized below.

IMHO reposting the entire driver makes reviewing easier.

And please address checkpatch and sparse complaints before reposting.
The code currently available at gitorious triggers several easy to
address checkpatch and spare complaints:
$ cd softiwarp
$ convert-file-to-patch *[ch] |
/usr/src/torvalds-2.6.git/scripts/checkpatch.pl - -notree -nosignoff |
grep -E '^ERROR|^WARNING' | grep -v LINUX_VERSION_CODE | sort | uniq
-c | sort -rn
     10 ERROR: spaces required around that '?' (ctx:VxV)
      8 ERROR: spaces required around that ':' (ctx:VxV)
      3 ERROR: spaces required around that ':' (ctx:VxW)
      2 ERROR: Macros with complex values should be enclosed in parenthesis
      1 WARNING: line over 80 characters
      1 ERROR: trailing whitespace
$ LC_ALL=C make C=2 | grep warning
/home/bart/software/softiwarp/softiwarp/siw_main.c:73:10: warning:
symbol 'siw_num_cep' was not declared. Should it be static?
/home/bart/software/softiwarp/softiwarp/siw_verbs.c:1417:14: warning:
symbol 'siw_reg_phys_mr' was not declared. Should it be static?
/home/bart/software/softiwarp/softiwarp/siw_mem.c:164:27: warning:
symbol 'siw_dma_mapping_ops' was not declared. Should it be static?

Bart.

Re: software iwarp stack update

[Roland Dreier]

On Wed, Jun 15, 2011 at 9:32 AM, Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org> wrote:
> IMHO reposting the entire driver makes reviewing easier.

Yes, I agree.  I don't think anyone is familiar enough with the
original version,
so an incremental patch is not particularly enlightening.

> And please address checkpatch and sparse complaints before reposting.

Right.  It's always best to take care of the trivial easy stuff first, to let
people focus on real issues.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bart Van Assche]

On Wed, Jun 15, 2011 at 5:43 PM, Bernard Metzler <BMT-OA+xvbQnYDHMbYB6QlFGEg@public.gmane.org> wrote:
> after rebasing to davem/net-next-2.6.git, I formatted a patch adding
> to the initial 'siw' driver posting from last October. Given the rather
> substantial amount of changes and time elapsed, I am not sure if people
> would better like a complete re-posting of all files.
> If I don't hear objection before tomorrow I would post it as an
> increment - as summarized below.

Documentation in general and for sysfs attributes in particular seems
to be missing ?

Also, the following statement in siw_main.c:

ibdev->modify_port = NULL;

causes the following behavior if ib_modify_port() is invoked on a
softiwarp port:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<          (null)>]           (null)
Call Trace:
 [<ffffffffa0313c15>] ? ib_modify_port+0x55/0x60 [ib_core]
[ ... ]

Shouldn't the siw driver return an error code instead of setting
ibdev->modify_port to NULL ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bernard Metzler]

linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org wrote on 06/15/2011 08:57:38 PM:

> On Wed, Jun 15, 2011 at 5:43 PM, Bernard Metzler <BMT-OA+xvbQnYDHMbYB6QlFGEg@public.gmane.org>
wrote:
> > after rebasing to davem/net-next-2.6.git, I formatted a patch adding
> > to the initial 'siw' driver posting from last October. Given the rather
> > substantial amount of changes and time elapsed, I am not sure if people
> > would better like a complete re-posting of all files.
> > If I don't hear objection before tomorrow I would post it as an
> > increment - as summarized below.
>
> Documentation in general and for sysfs attributes in particular seems
> to be missing ?
>
> Also, the following statement in siw_main.c:
>
> ibdev->modify_port = NULL;

oh, thanks...!

I was expecting there is an ib_device->uverbs_cmd_mask
flag which siw does not set... Unfortunately, it is not
completely clear to me when a command mask bit is available
and when the OFA core strictly expects a function pointer.
Maybe there is some redundancy/inconsistency?


Thanks,
Bernard.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Software iWARP Stack - take 2

[Bernard Metzler]

Bart, Roland, all,

many thanks for the suggestions and input. I am happy
to see interest in SoftiWARP.

In the hope to contribute something usefull, I will now
send a complete new set of patches against net-next.
It adds software iWARP support on top of TCP kernel sockets
and fits into the linux-rdma kernel stack.

Major changes to the previous (first) posting from October last
year include some kernel client support, restructuring of the
connection management code and general code cleanup.
I hope all issues which were rised after the first posting are
resolved. Thanks again for the valuable feedback.

Please take the current code contribution as work in progress.
Any comment and input is highly appreciated.


Here comes a shortlog of what you should expect:

Bernard Metzler (14):
      SIWv2: Kconfig and Makefile
      SIWv2: iWARP Protocol headers: iwarp.h
      SIWv2: Main header file: siw.h
      SIWv2: Module initialization: siw_main.c
      SIWv2: User interface: siw_verbs.h, siw_verbs.c, siw_user.h, siw_ae.c
      SIWv2: Connection management: siw_cm.c, siw_cm.h
      SIWv2: Object management: siw_obj.c, siw_obj.h
      SIWv2: Queue pair: siw_qp.c
      SIWv2: Completion queue: siw_cq.c
      SIWv2: Transmit path: siw_qp_tx.c
      SIWv2: Receive path: siw_qp_rx.c
      SIWv2: Debugging and Tracing: siw_debug.c, siw_debug.h
      SIWv2: Memory management: siw_mem.c
      SIWv2: Documentation: siw.txt



Many thanks,
Bernard.

Re: software iwarp stack update

[Bart Van Assche]

On Thu, Jun 16, 2011 at 1:51 PM, Bernard Metzler <BMT@zurich.ibm.com> wrote:
> > Also, the following statement in siw_main.c:
> >
> > ibdev->modify_port = NULL;
>
> I was expecting there is an ib_device->uverbs_cmd_mask
> flag which siw does not set... Unfortunately, it is not
> completely clear to me when a command mask bit is available
> and when the OFA core strictly expects a function pointer.
> Maybe there is some redundancy/inconsistency?

If other iWARP drivers (amso, cxgb3) have a modify_port method, why
doesn't the siw driver have such a method ? I see this as a blocking
issue.

Bart.

Re: software iwarp stack update

[Roland Dreier]

> I was expecting there is an ib_device->uverbs_cmd_mask
> flag which siw does not set... Unfortunately, it is not
> completely clear to me when a command mask bit is available
> and when the OFA core strictly expects a function pointer.
> Maybe there is some redundancy/inconsistency?

The uverbs_cmd_mask controls which commands userspace
can send down.  A kernel consumer does not go through the
code that checks this mask.

It does seem we are missing an IB_MANDATORY_FUNC
entry for modify_port in ib_device_check_mandatory; or on
the flip side we are missing a check of modify_port and a
-ENOSYS return... I guess modify_port does not really make
sense for iWARP so probably the second option is better.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bart Van Assche]

On Thu, Jun 16, 2011 at 7:20 PM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> It does seem we are missing an IB_MANDATORY_FUNC
> entry for modify_port in ib_device_check_mandatory; or on
> the flip side we are missing a check of modify_port and a
> -ENOSYS return... I guess modify_port does not really make
> sense for iWARP so probably the second option is better.

It's not just ib_modify_port() - there are several other functions in
the amso and cxgb* drivers that do nothing more than returning either
-ENOSYS or ERR_PTR(-ENOSYS).

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bernard Metzler]

Ok, let me put in all the -ENOSYS functions.
I did not know about that distinction between user
and kernel level clients.

Thanks,
Bernard.

linux-rdma-owner@vger.kernel.org wrote on 06/16/2011 08:06:44 PM:

> On Thu, Jun 16, 2011 at 7:20 PM, Roland Dreier <roland@purestorage.com>
wrote:
> > It does seem we are missing an IB_MANDATORY_FUNC
> > entry for modify_port in ib_device_check_mandatory; or on
> > the flip side we are missing a check of modify_port and a
> > -ENOSYS return... I guess modify_port does not really make
> > sense for iWARP so probably the second option is better.
>
> It's not just ib_modify_port() - there are several other functions in
> the amso and cxgb* drivers that do nothing more than returning either
> -ENOSYS or ERR_PTR(-ENOSYS).
>
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bart Van Assche]

On Thu, Jun 16, 2011 at 7:20 PM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> It does seem we are missing an IB_MANDATORY_FUNC
> entry for modify_port in ib_device_check_mandatory; or on
> the flip side we are missing a check of modify_port and a
> -ENOSYS return... I guess modify_port does not really make
> sense for iWARP so probably the second option is better.

There seems to be disagreement about whether to return 0, -ENOSYS or
-EOPNOTSUPP for not supported functionality. Does the patch below make sense ?
Note: I don't have all the hardware necessary to test the patch below.

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 4007f72..e711de4 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -627,6 +627,9 @@ int ib_modify_device(struct ib_device *device,
 		     int device_modify_mask,
 		     struct ib_device_modify *device_modify)
 {
+	if (!device->modify_device)
+		return -ENOSYS;
+
 	return device->modify_device(device, device_modify_mask,
 				     device_modify);
 }
@@ -647,6 +650,9 @@ int ib_modify_port(struct ib_device *device,
 		   u8 port_num, int port_modify_mask,
 		   struct ib_port_modify *port_modify)
 {
+	if (!device->modify_port)
+		return -ENOSYS;
+
 	if (port_num < start_port(device) || port_num > end_port(device))
 		return -EINVAL;

diff --git a/drivers/infiniband/hw/amso1100/c2_provider.c
b/drivers/infiniband/hw/amso1100/c2_provider.c
index aeebc4d..f101bb7 100644
--- a/drivers/infiniband/hw/amso1100/c2_provider.c
+++ b/drivers/infiniband/hw/amso1100/c2_provider.c
@@ -99,14 +99,6 @@ static int c2_query_port(struct ib_device *ibdev,
 	return 0;
 }

-static int c2_modify_port(struct ib_device *ibdev,
-			  u8 port, int port_modify_mask,
-			  struct ib_port_modify *props)
-{
-	pr_debug("%s:%u\n", __func__, __LINE__);
-	return 0;
-}
-
 static int c2_query_pkey(struct ib_device *ibdev,
 			 u8 port, u16 index, u16 * pkey)
 {
@@ -817,7 +809,6 @@ int c2_register_device(struct c2_dev *dev)
 	dev->ibdev.dma_device = &dev->pcidev->dev;
 	dev->ibdev.query_device = c2_query_device;
 	dev->ibdev.query_port = c2_query_port;
-	dev->ibdev.modify_port = c2_modify_port;
 	dev->ibdev.query_pkey = c2_query_pkey;
 	dev->ibdev.query_gid = c2_query_gid;
 	dev->ibdev.alloc_ucontext = c2_alloc_ucontext;
diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c
b/drivers/infiniband/hw/cxgb3/iwch_provider.c
index 2e27413..c7d9411 100644
--- a/drivers/infiniband/hw/cxgb3/iwch_provider.c
+++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c
@@ -61,13 +61,6 @@
 #include "iwch_user.h"
 #include "common.h"

-static int iwch_modify_port(struct ib_device *ibdev,
-			    u8 port, int port_modify_mask,
-			    struct ib_port_modify *props)
-{
-	return -ENOSYS;
-}
-
 static struct ib_ah *iwch_ah_create(struct ib_pd *pd,
 				    struct ib_ah_attr *ah_attr)
 {
@@ -1392,7 +1385,6 @@ int iwch_register_device(struct iwch_dev *dev)
 	dev->ibdev.dma_device = &(dev->rdev.rnic_info.pdev->dev);
 	dev->ibdev.query_device = iwch_query_device;
 	dev->ibdev.query_port = iwch_query_port;
-	dev->ibdev.modify_port = iwch_modify_port;
 	dev->ibdev.query_pkey = iwch_query_pkey;
 	dev->ibdev.query_gid = iwch_query_gid;
 	dev->ibdev.alloc_ucontext = iwch_alloc_ucontext;
diff --git a/drivers/infiniband/hw/cxgb4/provider.c
b/drivers/infiniband/hw/cxgb4/provider.c
index 5b9e422..247fe70 100644
--- a/drivers/infiniband/hw/cxgb4/provider.c
+++ b/drivers/infiniband/hw/cxgb4/provider.c
@@ -58,13 +58,6 @@ static int fastreg_support = 1;
 module_param(fastreg_support, int, 0644);
 MODULE_PARM_DESC(fastreg_support, "Advertise fastreg support (default=1)");

-static int c4iw_modify_port(struct ib_device *ibdev,
-			    u8 port, int port_modify_mask,
-			    struct ib_port_modify *props)
-{
-	return -ENOSYS;
-}
-
 static struct ib_ah *c4iw_ah_create(struct ib_pd *pd,
 				    struct ib_ah_attr *ah_attr)
 {
@@ -456,7 +449,6 @@ int c4iw_register_device(struct c4iw_dev *dev)
 	dev->ibdev.dma_device = &(dev->rdev.lldi.pdev->dev);
 	dev->ibdev.query_device = c4iw_query_device;
 	dev->ibdev.query_port = c4iw_query_port;
-	dev->ibdev.modify_port = c4iw_modify_port;
 	dev->ibdev.query_pkey = c4iw_query_pkey;
 	dev->ibdev.query_gid = c4iw_query_gid;
 	dev->ibdev.alloc_ucontext = c4iw_alloc_ucontext;
diff --git a/drivers/infiniband/hw/nes/nes_verbs.c
b/drivers/infiniband/hw/nes/nes_verbs.c
index 95ca93c..9f2f7d4 100644
--- a/drivers/infiniband/hw/nes/nes_verbs.c
+++ b/drivers/infiniband/hw/nes/nes_verbs.c
@@ -605,16 +605,6 @@ static int nes_query_port(struct ib_device
*ibdev, u8 port, struct ib_port_attr


 /**
- * nes_modify_port
- */
-static int nes_modify_port(struct ib_device *ibdev, u8 port,
-		int port_modify_mask, struct ib_port_modify *props)
-{
-	return 0;
-}
-
-
-/**
  * nes_query_pkey
  */
 static int nes_query_pkey(struct ib_device *ibdev, u8 port, u16
index, u16 *pkey)
@@ -3882,7 +3872,6 @@ struct nes_ib_device *nes_init_ofa_device(struct
net_device *netdev)
 	nesibdev->ibdev.dev.parent = &nesdev->pcidev->dev;
 	nesibdev->ibdev.query_device = nes_query_device;
 	nesibdev->ibdev.query_port = nes_query_port;
-	nesibdev->ibdev.modify_port = nes_modify_port;
 	nesibdev->ibdev.query_pkey = nes_query_pkey;
 	nesibdev->ibdev.query_gid = nes_query_gid;
 	nesibdev->ibdev.alloc_ucontext = nes_alloc_ucontext;
diff --git a/drivers/infiniband/hw/siw/siw_main.c
b/drivers/infiniband/hw/siw/siw_main.c
index b2b7bea..d2346fe 100644
--- a/drivers/infiniband/hw/siw/siw_main.c
+++ b/drivers/infiniband/hw/siw/siw_main.c
@@ -216,13 +216,6 @@ static struct device_attribute *siw_dev_attributes[] = {
 	&dev_attr_cep
 };

-static int siw_modify_port(struct ib_device *ofa_dev, u8 port, int mask,
-			   struct ib_port_modify *props)
-{
-	return -EOPNOTSUPP;
-}
-
-
 static int siw_register_device(struct siw_dev *dev)
 {
 	struct ib_device *ibdev = &dev->ofa_dev;
@@ -284,7 +277,6 @@ static int siw_register_device(struct siw_dev *dev)
 	ibdev->query_device = siw_query_device;
 	ibdev->query_port = siw_query_port;
 	ibdev->query_qp = siw_query_qp;
-	ibdev->modify_port = siw_modify_port;
 	ibdev->query_pkey = siw_query_pkey;
 	ibdev->query_gid = siw_query_gid;
 	ibdev->alloc_ucontext = siw_alloc_ucontext;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Or Gerlitz]

Bart Van Assche wrote:
> Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
>> It does seem we are missing an IB_MANDATORY_FUNC
>> entry for modify_port in ib_device_check_mandatory; or on
>> the flip side we are missing a check of modify_port and a
>> -ENOSYS return... I guess modify_port does not really make
>> sense for iWARP so probably the second option is better.
> 
> There seems to be disagreement about whether to return 0, -ENOSYS or
> -EOPNOTSUPP for not supported functionality. Does the patch below make sense ?
> Note: I don't have all the hardware necessary to test the patch below.

Bernard, the netdev guys have enough on their plate... what's the reason 
we want this thread the be cross posted to both linux-rdma and netdev?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Bernard Metzler]

Hi Or,
I understand your point.

Main reason for having netdev on cc is that I think adding
softiwarp to the kernel is adding a service to the kernel
network stack. If accepted, iWARP as a protocol would become a 
Linux kernel service, and it would directly make use of 
kernel TCP/IP services. That is different form all other
rdma verbs providers, which offload the data path to private
territory.

At the current point, maybe its reasonable to restrict
detailed discussion to linux-rdma, if only issues related to the
linux-rdma stack are discussed? I would be happy to keep netdev
in the loop for patches and of course for any discussion touching
softiwarp's interaction with network kernel services.

Many thanks,
Bernard.

Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote on 06/19/2011 07:12:14 AM:

> Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 
> 06/19/2011 07:12 AM
> 
> To
> 
> Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>, Bernard Metzler 
<BMT-OA+xvbQnYDHMbYB6QlFGEg@public.gmane.org>
> 
> cc
> 
> Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org>, <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> 
> Subject
> 
> Re: software iwarp stack update
> 
> Bart Van Assche wrote:
> > Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> >> It does seem we are missing an IB_MANDATORY_FUNC
> >> entry for modify_port in ib_device_check_mandatory; or on
> >> the flip side we are missing a check of modify_port and a
> >> -ENOSYS return... I guess modify_port does not really make
> >> sense for iWARP so probably the second option is better.
> > 
> > There seems to be disagreement about whether to return 0, -ENOSYS or
> > -EOPNOTSUPP for not supported functionality. Does the patch below 
> make sense ?
> > Note: I don't have all the hardware necessary to test the patch below.
> 
> Bernard, the netdev guys have enough on their plate... what's the reason 

> we want this thread the be cross posted to both linux-rdma and netdev?
> 
> Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: software iwarp stack update

[Roland Dreier]

> There seems to be disagreement about whether to return 0, -ENOSYS or
> -EOPNOTSUPP for not supported functionality. Does the patch below make sense ?
> Note: I don't have all the hardware necessary to test the patch below.

This makes sense to me, so I went ahead and applied it for 3.1 (minus
the siw part, of course).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html