Re: PATCH - InfiniBand Access Layer (IBAL)

Re: PATCH - InfiniBand Access Layer (IBAL)

[Nivedita Singhvi]

> That is, of course, an excellent approach.
> 
> But beware of being *too* disconnected from the lists@vger.kernel.org.  We
> don't want to get in the situation where you pop up with a couple of
> person-years' worth of work and other kernel developers have major issues
> with it.  Please find a balance - some way of regularly checkpointing.

Andrew,

Thanks. I had wanted to convey the gist of what you say to someone
earlier this week.  It would have been nice to point to some document
that captures this, and though I looked around, I haven't found anything
in one place, surprisingly. It's a plea kernel maintainers frequently make
to Linux contributors, but new projects seem to run into the same problems
repeatedly. I figured it would be worth putting together your comments
and some basic related advice given to those wishing to get their
code into the kernel in a document. Hope that's ok.

File follows in a separate mail.

thanks,
Nivedita

Re: PATCH - InfiniBand Access Layer (IBAL)

[Roland Dreier]

    Ulrich> The only acceptable order in which things can happen is:

    Ulrich> 1. develop API
    Ulrich> 2. propose API to be accepted by "community"/distributions
    Ulrich> 3. change API if necessary, and go back to 2.
    Ulrich> 4. write applications using new API

I don't think this is reasonable, since nothing is settled enough for
this to work.  On the one hand, InfiniBand and other "fibers" (eg RDMA
over ethernet) are quite experimental.  No one is sure of the right
semantics or the best way to use the interconnect.  On the other hand,
there are people who want to use this stuff right now (eg
high-performance computing people building clusters, database cluster
people, etc).

There are users who want to use InfiniBand now, and making them wait
through your whole process above is simply untenable.  You can't
expect a company selling InfiniBand equipment to say, "Sorry, our
software isn't perfect (although it would work for you now).  Come
back in a year or two."

With that in mind, I think the only order things can happen is:

    1. develop API
    2. implement API
    2a.learn from mistakes and go back to 1.
    3. write applications using API
    4. learn from mistakes and go back to 1.

It's certainly unfortunate that so much InfiniBand software has been
developed behind closed doors, but the industry has finally woken up
and come together around the OpenIB idea to develop Linux support
completely in the open.

When does this software make it into distributions?  Obviously that's
up to the distribution.  Certainly a commercial distribution has
customers of its own to listen to, and I would assume that the
decision would be made based on the appropriate combination of
technical merit and customer demand.

 - Roland

RE: PATCH - InfiniBand Access Layer (IBAL)

[Acker, Dave]

> -----Original Message-----
> From: Ulrich Drepper [mailto:drepper@redhat.com]
> Sent: Saturday, March 20, 2004 12:56 PM
> To: Acker, Dave
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: PATCH - InfiniBand Access Layer (IBAL)
> 
> Acker, Dave wrote:
> > I can say that these other APIs are
> > already being used by other programs (or other APIs themselves) and
> > really can't go away.  If folks ask for ICSC support, it will
probably
> > get in there.
> 
> Spoken in true "proprietary-software, don't get the concept of free
> software"-fashion.
> 
> Just consider the implications if this would be accepted as an
argument.
>  Everybody who is producing a new API has to create just a couple of
> programs using it before publishing the specs to be able to say: well,
> here's the API.  Use it.  Oh, and don't change it, that's not
acceptable
> because we have code relying on the API.  Yep, strong-arming people
you
> have no control over works out fine, all the time.
> 
> The only acceptable order in which things can happen is:
> 
> 1. develop API
> 
> 2. propose API to be accepted by "community"/distributions
> 
> 3. change API if necessary, and go back to 2.
> 
> 4. write applications using new API
> 
> 

I don't mean that these other APIs MUST be accepted...just that people
are using them and will want to keep using them because they work for
them.  Even if there is one true standard linux interconnect API, I
suspect this APIs may hang around outside of standard linux.  Changing
an API is a lot easier than changing all the software that uses it.
 
I mention these other APIS because people have an interest in them
already and having interest in an API is a good thing.  People who are
actually going to use the API bring useful comments to the table.
Developing an API that no one wants to use isn't a very good way to
spend your time in my opinion.
The process you define is great if a company can wait for the process to
shake out a final API.  Often companies can not wait and have to move
forward.  The IB companies did just that.  This isn't to say we are not
open to moving from a proprietary API we made out of need to a community
accepted one.  That is what OpenIB.org is all about.  If we didn't care
we would not have bothered with this.  We are not trying to "strong-arm"
anyone into anything.  We want an open standard and are willing to
follow this process.  All of the companies involved have agreed to
support the results of this process. 


> > If folks ask for ICSC support, it will probably
> > get in there.
> 
> You did not in the least address the main point: IB is just one fiber.
> I cannot imagine anybody but the IB people want to have a specific API
+
> kernel extensions for each separate interconnect fiber.
> 
> Get it all over with in one stroke.  Come up with an API which covers
it
> all.  The requirements aren't that different.  And I singled out ICSC
> because it attempts to do just this.
> 

We are not looking for a specific API + kernel extensions for each
interconnect fiber.  We work in the area of IB and want to solve its
issues but I don't think that we are against a standard that solves all
interconnect issues.  I don't believe that we are against an API that
covers it all.  We (most IB companies) have supported DAPL and MPI.
Both are standards that try to work for all interconnects, not just IB.
If the community sees these as the wrong way to go and ICSC (or
something else) as the correct way then we (the IB community) will work
to support it.

> And don't say "we did our part, if the Linux community wants to have
> something else it's their to come up with a unified solution".  That
> would be acceptable only if it wouldn't be the IB people who want
their
> code to be generally accepted.  If you don't care about the later,
fine,
> leave it all as is.  But the code might not be picked up at all.
> 

Again, we are totally open to a unified solution...I don't believe we
have ever said we were not.  We started with looking for an InfiniBand
solution because that is what we know about and deal with everyday.

> Furthermore, don't count on much community participation.  There
aren't
> many people out there with the necessary hardware.  Or even the
ability
> to collect experiences.  The price for the changes have to be carried
by
> the parties with the agenda.
> 

Sure.  We expect to be making the changes ourselves often.

> 
> > If there is
> > going to be a standard linux infiniband stack it will have to be
very
> > good or else splinter versions and incompatible versions will live
on.
> 
> And by very good you mean of course your implementation.  From the
> comment above it is clear that if the standardized stack does not
> include your code you intent to keep using your code anyway.
Designing
> standards is always about compromises.
> 

Not at all.  In fact, the stack my company uses is not the base stack
openib.org has starting with, yet InfiniCon is still involved.
Compromises have already been made and more will be made in the future.
By "very good", I just mean that we want lots of review on the code.  We
want to hear what the community has to say.

> The organizations with interconnect interest have to come together to
> create just this, a compromise which in the end might not fulfill you
> "very good" criteria in all places.
> 

Sure.  It is totally possible that the final version lacks things that
some IB folks see as "very good" or needed.  This is where IB companies
can still offer and support additional pieces of open software that
suite their customer's needs.  For example, it is doubtful that MPI will
end up being the interconnect standard accepted by the linux community.
It is likely that its users will not be willing to change to another
interconnect API, even a standard one.  The rewrite cost would be too
high for many of them.  In these cases companies that sell interconnects
used for MPI applications will have to continue to support MPI.
Meanwhile others are free to use the standard out of the box and get
standard kernel support for it.

I am pretty sure RedHat and other Linux distributors have done similar
things for similar reasons.

-Ack

p.s. Again, these are my opinions, not the official opinions of
InfiniCon. 

Re: PATCH - InfiniBand Access Layer (IBAL)

[Ulrich Drepper]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Acker, Dave wrote:
> I can say that these other APIs are
> already being used by other programs (or other APIs themselves) and
> really can't go away.  If folks ask for ICSC support, it will probably
> get in there.

Spoken in true "proprietary-software, don't get the concept of free
software"-fashion.

Just consider the implications if this would be accepted as an argument.
 Everybody who is producing a new API has to create just a couple of
programs using it before publishing the specs to be able to say: well,
here's the API.  Use it.  Oh, and don't change it, that's not acceptable
because we have code relying on the API.  Yep, strong-arming people you
have no control over works out fine, all the time.

The only acceptable order in which things can happen is:

1. develop API

2. propose API to be accepted by "community"/distributions

3. change API if necessary, and go back to 2.

4. write applications using new API


> If folks ask for ICSC support, it will probably
> get in there.

You did not in the least address the main point: IB is just one fiber.
I cannot imagine anybody but the IB people want to have a specific API +
kernel extensions for each separate interconnect fiber.

Get it all over with in one stroke.  Come up with an API which covers it
all.  The requirements aren't that different.  And I singled out ICSC
because it attempts to do just this.

And don't say "we did our part, if the Linux community wants to have
something else it's their to come up with a unified solution".  That
would be acceptable only if it wouldn't be the IB people who want their
code to be generally accepted.  If you don't care about the later, fine,
leave it all as is.  But the code might not be picked up at all.

Furthermore, don't count on much community participation.  There aren't
many people out there with the necessary hardware.  Or even the ability
to collect experiences.  The price for the changes have to be carried by
the parties with the agenda.


> If there is
> going to be a standard linux infiniband stack it will have to be very
> good or else splinter versions and incompatible versions will live on.

And by very good you mean of course your implementation.  From the
comment above it is clear that if the standardized stack does not
include your code you intent to keep using your code anyway.  Designing
standards is always about compromises.

Nobody ever was/is happy with everything that POSIX requires.  But it is
an acceptable compromise.  One thing which has been clearly shown by
POSIX is that 99% of all developers prefer portability over getting the
last 5% of performance out of their machines.

The organizations with interconnect interest have to come together to
create just this, a compromise which in the end might not fulfill you
"very good" criteria in all places.

- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFAXIWg2ijCOnn/RHQRAk9OAKCPAanX45/cK3zdFEN/Y28gk/vHzwCdG57w
XprZ1vZdfkT8VY3CW6T+aR0=
=gYdV
-----END PGP SIGNATURE-----

Re: PATCH - InfiniBand Access Layer (IBAL)

[Acker, Dave]

Ulrich Drepper <drepper@redhat.com> wrote in message
news:<1ByNc-1km-21@gated-at.bofh.it>...
> 
> First, the working group I refer to is this:
> 
>   http://www.opengroup.org/icsc/
> 
> But that's all there is.  The socket extension working group
> 
>   http://www.opengroup.org/icsc/sockets/
> 

Most of us IB folks wanted to work with the ICSC stuff waaaaay back but
it never went anywhere and we all had to ship products.  So various
stacks developed.  All that is being specified by the ICSC is an API.
There are already several APIs into IB that are not IB specific.
These include:
DAPL - http://www.datcollaborative.org/,
http://sourceforge.net/projects/dapl/ 
MPI - http://www-unix.mcs.anl.gov/mpi/ 
SDP - (not really an API but is run over IB verbs and RDMA verbs and
meets up with your sockets note) See InfiniBand Specification, Volume 1,
Annex A4, Release 1.1, http://www.rdmaconsortium.org/home 

There is no reason we can't implement support for the ICSC APIs if the
Linux community desires this.  I can say that these other APIs are
already being used by other programs (or other APIs themselves) and
really can't go away.  If folks ask for ICSC support, it will probably
get in there.

> So, these people come up with their own software stacks, unreviewed
> interface extensions, and demand that everybody accepts what they were
> "designing" without the ability to question anything.
> I surely find this completely  unacceptable and any consideration of
> accepting anything the Infiniband group comes up with should be
> postponed until every bit of the design can be reviewed.  If bits and
> pieces are accepted prematurely it'll just be "now that this is
support
> you have to add this too, otherwise it'll not be useful".

While some of the same people were on the ICSC lists as the IBTA lists
or the DAT lists, it doesn't mean that they are operating as one big
group with some secret agenda.  The ICSC group went one way and only
recently produced a spec.  The DAT group went another and has had a spec
for some time.  The RDMA folks went yet another and produced a spec a
bit ago.  While there are some people in both groups, the groups act
independently with different companies and individuals have more
control/influence in one than the other.

All that said, the IB developer community has decided as a whole to open
things up.  InfiniCon, TopSpin, Voltaire, DivergeNet, Mellanox, etc.
have all opened up their code.  Add to this the already started open
effort on sourceforge (IBAL) and we hope to find a best of breed final
solution.  So please do review every single bit posted.  If there is
going to be a standard linux infiniband stack it will have to be very
good or else splinter versions and incompatible versions will live on.

-Ack

p.s. While I may work for an IB company, I am expressing my own
opinions, not their official opinions.

Re: PATCH - InfiniBand Access Layer (IBAL)

[Roland Dreier]

    Ulrich> I think I should be telling something about another nuance
    Ulrich> of this problem.  Parties interested in Infiniband have
    Ulrich> been working under the OpenGroup umbrella for quite some
    Ulrich> time now on API extensions to better accommodate
    Ulrich> interconnect fibers.  They've even presented at am Austin
    Ulrich> Group meeting in 2001 (I think) to get on the road map for
    Ulrich> being included in POSIX.

    Ulrich> But when I wanted to take a look at the specs this was
    Ulrich> categorically rejected.  My contacts were explicitly
    Ulrich> forbidden to give the drafts to anybody but the elite
    Ulrich> circle.  Mind you, Red Hat is member of the OpenGroup.

    Ulrich> So, these people come up with their own software stacks,
    Ulrich> unreviewed interface extensions, and demand that everybody
    Ulrich> accepts what they were "designing" without the ability to
    Ulrich> question anything.

    Ulrich> I surely find this completely unacceptable and any
    Ulrich> consideration of accepting anything the Infiniband group
    Ulrich> comes up with should be postponed until every bit of the
    Ulrich> design can be reviewed.  If bits and pieces are accepted
    Ulrich> prematurely it'll just be "now that this is support you
    Ulrich> have to add this too, otherwise it'll not be useful".

I believe what you are referring to is the OpenGroup's "ITAPI" work.
This is in a sense a competing spec to the DAT Collaborative
(www.datcollaborative.org) work.  The DAT Collaborative seems to be
far more open, and their spec is available without any hurdles.

Any demands from groups designing unacceptable specs should be treated
with the appropriate level of cooperation.

Note that neither the OpenGroup nor the DAT Collaborative are
affiliated with the InfiniBand Trade Association or the OpenIB group,
although they may have members in common.

Best,
  Roland

Re: PATCH - InfiniBand Access Layer (IBAL)

[Ulrich Drepper]

> For the IBAL stack, there are numerous documents on the Linux InfiniBand
> Project (http://infiniband.sourceforge.net/) describing most everything from
> the overall architecture to the APIs.

That's not the same, and it incidentally shows a problem in what you do.

First, the working group I refer to is this:

  http://www.opengroup.org/icsc/

Dave Edmondson from Sun gave a presentation at the Austin group meeting
(in 2002, not 2001 as I said before) which you can see here:

  http://www.opengroup.org/austin/docs/austin_105.pdf


This is the information I cannot get to.  Now why would I want that
instead of what you do?

First of all, these are the extensions to the existing interfaces.
These extensions not only influence existing code, but the extensions
can also be useful for non-interconnect usage.  That is, if we can adopt
them if necessary.  Many of the interconnect problems also are present
in Gig and 10Gig ethernet.  The ICSC refused to give anyone outside
their group access to the document despite the fact that they want to
have the extensions added to POSIX.  If we develop separate extensions
they will collide and/or cause unnecessary duplication of code and effort.


Now, why is ICSC relevant?  In my opinion, and I'm not really that
knowledgeable about all these networking issues but I know a thing or
two about APIs, going down the road with Infiniband specific APIs is
bad.  Why would we want to have separate APIs for other interconnect
fibers?  The ICSC, according to my understanding, tries to unify the
APIs to be usable not only by Infiniband.  This is IMO highly desirable.

You can get a glimpse of what they are doing by looking at the documents
referenced in

  http://www.opengroup.org/icsc/

But that's all there is.  The socket extension working group

  http://www.opengroup.org/icsc/sockets/

only has some meeting minutes available.

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖

RE: PATCH - InfiniBand Access Layer (IBAL)

[Fab Tillier]

> -----Original Message-----
> From: Ulrich Drepper [mailto:drepper@redhat.com]
> Sent: Friday, March 19, 2004 10:47 AM
> 
> So, these people come up with their own software stacks, unreviewed
> interface extensions, and demand that everybody accepts what they were
> "designing" without the ability to question anything.

Yes, a design review with a period to provide feedback at the design level,
not the code level, would make sense.  I don't see how one could argue
against that.

> 
> I surely find this completely  unacceptable and any consideration of
> accepting anything the Infiniband group comes up with should be
> postponed until every bit of the design can be reviewed.  If bits and
> pieces are accepted prematurely it'll just be "now that this is support
> you have to add this too, otherwise it'll not be useful".

For the IBAL stack, there are numerous documents on the Linux InfiniBand
Project (http://infiniband.sourceforge.net/) describing most everything from
the overall architecture to the APIs.  On the project home page is a general
overview of what InfiniBand is, and how it fits into the OS.  More detailed
documentation is available there too.  Of particular interest to this thread
would be the Access Layer documents. Below are links to documents of
interest.

- The overall software architecture spec is the "Linux SAS", available at
http://infiniband.sourceforge.net/LinuxSAS.1.0.1.pdf.
- A presentation describing the IBAL APIs is here:
http://infiniband.sourceforge.net/IAL/Access/AlInterface.pdf
- The IBAL high level design is here:
http://infiniband.sourceforge.net/IAL/Access/IBA_AL_HLD.pdf
- A user's guide to IBAL is here:
http://infiniband.sourceforge.net/IAL/Access/AL_Users_Guide.pdf
- And finally, the API documentation is here:
http://infiniband.sourceforge.net/IAL/Access/IBAL/IBAL_mi.html

HTH,

- Fab

Re: PATCH - InfiniBand Access Layer (IBAL)

[Ulrich Drepper]

I think I should be telling something about another nuance of this problem.


Parties interested in Infiniband have been working under the OpenGroup
umbrella for quite some time now on API extensions to better accommodate
interconnect fibers.  They've even presented at am Austin Group meeting
in 2001 (I think) to get on the road map for being included in POSIX.

But when I wanted to take a look at the specs this was categorically
rejected.  My contacts were explicitly forbidden to give the drafts to
anybody but the elite circle.  Mind you, Red Hat is member of the OpenGroup.

So, these people come up with their own software stacks, unreviewed
interface extensions, and demand that everybody accepts what they were
"designing" without the ability to question anything.

I surely find this completely  unacceptable and any consideration of
accepting anything the Infiniband group comes up with should be
postponed until every bit of the design can be reviewed.  If bits and
pieces are accepted prematurely it'll just be "now that this is support
you have to add this too, otherwise it'll not be useful".

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖

Re: PATCH - InfiniBand Access Layer (IBAL)

[Roland Dreier]

I understand that you've only had a short time to review the code, and
many of your comments are points well taken.  I think most of the
technical comments can wait to be debated on the openib.org mailing
lists (which I am assured are coming soon).  However, since this is
being preserved for posterity in the linux-kernel archive, I wanted to
correct a few inaccuracies.

    Robert> 3.) The code is not compliant with the InfiniBand
    Robert> specification and has proprietary implementations of
    Robert> things like "path records" so it will only work with the
    Robert> TopSpin subnet manager that requires you to buy a topspin
    Robert> switch.

This is definitely not our intent, and we fix whatever IB compliancy
bugs we find as soon as we know about them.  In addition, I know that
people have been able to use the Topspin/Mellanox code from OpenIB
with OpenSM and no Topspin switch (this did require some compliance
fixes to both OpenSM and the Topspin code).

    Robert> 6.)The VAPI code has extra propietary verbs that are not
    Robert> specified by the InfiniBand Specification.

True, but I'm not sure why this is a deficiency.  We found certain
things that were required for good performance were not specified by
the IBTA, and we added them.  The Linux way is definitely not to
follow a spec when the spec is wrong.

    Robert> 9.) The CM does not support reliable datagrams.

Fair enough but I'm sure we can easily add RD support before any
hardware supporting RD is ready.  We took a pragmatic approach and
didn't implement "speculative" features.

    Robert> 10) There is no built in support for plug and play events,
    Robert> port up/down, LID change, SM change

I'm not sure what plug and play events are (certainly they're not part
of the IB spec).  However, we did add extended IB asynchronous events
for LID/SM changes and P_Key changes (port up and down are already
part of the IB spec).

As I said above, the rest of your points are well taken (although of
course there are two sides to every story) and we can talk about them
when we get the openib.org mailing lists up.

 - Roland

Re: PATCH - InfiniBand Access Layer (IBAL)

[Johannes Erdfelt]

On Tue, Mar 16, 2004, Christoph Hellwig <hch@infradead.org> wrote:
> On Mon, Mar 15, 2004 at 03:54:14PM -0800, Johannes Erdfelt wrote:
> > > Did you actually read it?
> > 
> > The code on openib.org? Yes, I wrote some of it.
> > 
> > I would be the first to say that there are portions that need to be
> > rewritten, but I definately do not think all or even most of it does.
> > 
> > That's why I was asking what specifically you found fatally wrong with
> > it. I haven't seen many critiques, so I can only assume it's the same
> > things I see wrong with it.
> 
> Start with the thing Robert already mentioned.  Ad ontop of that:
> 
>  - the horrible Winodes/Linux compat code.  We all know this kind
>    of compat code is messy.  But the way it's don in this code is just
>    incredibly idiotic.
>  - totally braindead use of macro abstraction
>  - those split into far too many files
>  - wrong use of dma mapping abstraction
>  - braindead malloc code
>  - wrong modversions handling duplicated in every file
> 
> I'm really surprised you're admitting to having touched that code.
> I'd have guessed everyone who did would hide in his house ashamed.

You do realize that the code on openib.org is from multiple vendors,
right? I only touched one part of that code. That's why I said 'some'.

Only some of the code has the problems you listed, and some of those are
far from fatal.

How about I ask you what parts of the code do you feel don't need a
complete rewrite?

> > > p.s.  if you reply to my mails please keep me in the To line.  Really,
> > > please don't do any fany reply to group tricks unless people explicitly
> > > request it in the Mail-Fup header.
> > 
> > If you really want duplicates of all the replies, sure, I'll make an
> > exception for you.
> > 
> > I don't see why a smarter client, or mail filter, couldn't do the same
> > thing without depending on the behaviour of the sender.
> 
> Replying to people personally is good taste.  You might know I'm on
> lkml but on many other lists I'm not.  As are other people on lkml.
> A filter can easily filter out duplicates but it can't magically
> create copies of mails not addressed to you. 

Sure it can. I replied to your email. Check the mail headers,
specifically the ones labeled References and In-Reply-To.

> In addition I tend to read my inbox fairly quick all the time and
> the lkml mailbox only when I'm at least a little idle.

I didn't need an immediate answer.

JE

Re: PATCH - InfiniBand Access Layer (IBAL)

[Christoph Hellwig]

On Mon, Mar 15, 2004 at 03:54:14PM -0800, Johannes Erdfelt wrote:
> > Did you actually read it?
> 
> The code on openib.org? Yes, I wrote some of it.
> 
> I would be the first to say that there are portions that need to be
> rewritten, but I definately do not think all or even most of it does.
> 
> That's why I was asking what specifically you found fatally wrong with
> it. I haven't seen many critiques, so I can only assume it's the same
> things I see wrong with it.

Start with the thing Robert already mentioned.  Ad ontop of that:

 - the horrible Winodes/Linux compat code.  We all know this kind
   of compat code is messy.  But the way it's don in this code is just
   incredibly idiotic.
 - totally braindead use of macro abstraction
 - those split into far too many files
 - wrong use of dma mapping abstraction
 - braindead malloc code
 - wrong modversions handling duplicated in every file

I'm really surprised you're admitting to having touched that code.
I'd have guessed everyone who did would hide in his house ashamed.

> > p.s.  if you reply to my mails please keep me in the To line.  Really,
> > please don't do any fany reply to group tricks unless people explicitly
> > request it in the Mail-Fup header.
> 
> If you really want duplicates of all the replies, sure, I'll make an
> exception for you.
> 
> I don't see why a smarter client, or mail filter, couldn't do the same
> thing without depending on the behaviour of the sender.

Replying to people personally is good taste.  You might know I'm on
lkml but on many other lists I'm not.  As are other people on lkml.
A filter can easily filter out duplicates but it can't magically
create copies of mails not addressed to you. 

In addition I tend to read my inbox fairly quick all the time and
the lkml mailbox only when I'm at least a little idle.

Re: PATCH - InfiniBand Access Layer (IBAL)

[Johannes Erdfelt]

On Mon, Mar 15, 2004, Christoph Hellwig <hch@infradead.org> wrote:
> On Mon, Mar 15, 2004 at 03:44:25PM -0800, Johannes Erdfelt wrote:
> > > From looking at both codebases starting from scratch sounds like the
> > > best idea to me..  
> > 
> > What's fatally wrong with the code that's currently available via
> > openib.org?
> 
> Did you actually read it?

The code on openib.org? Yes, I wrote some of it.

I would be the first to say that there are portions that need to be
rewritten, but I definately do not think all or even most of it does.

That's why I was asking what specifically you found fatally wrong with
it. I haven't seen many critiques, so I can only assume it's the same
things I see wrong with it.

> p.s.  if you reply to my mails please keep me in the To line.  Really,
> please don't do any fany reply to group tricks unless people explicitly
> request it in the Mail-Fup header.

If you really want duplicates of all the replies, sure, I'll make an
exception for you.

I don't see why a smarter client, or mail filter, couldn't do the same
thing without depending on the behaviour of the sender.

JE

Re: PATCH - InfiniBand Access Layer (IBAL)

[Christoph Hellwig]

On Mon, Mar 15, 2004 at 03:44:25PM -0800, Johannes Erdfelt wrote:
> > From looking at both codebases starting from scratch sounds like the
> > best idea to me..  
> 
> What's fatally wrong with the code that's currently available via
> openib.org?

Did you actually read it?

p.s.  if you reply to my mails please keep me in the To line.  Really,
please don't do any fany reply to group tricks unless people explicitly
request it in the Mail-Fup header.

Re: PATCH - InfiniBand Access Layer (IBAL)

[Johannes Erdfelt]

On Mon, Mar 15, 2004, Christoph Hellwig <hch@infradead.org> wrote:
> On Mon, Mar 15, 2004 at 02:52:44PM -0800, Woodruff, Robert J wrote:
> > As I stated above, we are part of the openib.org collaboration and 
> > will be working on helping develop a stack that is "best of
> > breed" from all of the available code. Starting from the bottom up, 
> > we first need to review the various proposals for the 
> > Access Layer and determine which code base to start with. 
> 
> From looking at both codebases starting from scratch sounds like the
> best idea to me..  

What's fatally wrong with the code that's currently available via
openib.org?

JE

Re: PATCH - InfiniBand Access Layer (IBAL)

[Christoph Hellwig]

On Mon, Mar 15, 2004 at 02:52:44PM -0800, Woodruff, Robert J wrote:
> As I stated above, we are part of the openib.org collaboration and 
> will be working on helping develop a stack that is "best of
> breed" from all of the available code. Starting from the bottom up, 
> we first need to review the various proposals for the 
> Access Layer and determine which code base to start with. 

>From looking at both codebases starting from scratch sounds like the
best idea to me..  

RE: PATCH - InfiniBand Access Layer (IBAL)

[Woodruff, Robert J]

On Sun, Mar 14, Greg KH wrote:

First, As my boss remined me this morning,
let me make sure I was clear, there are not 2 different efforts now,
only one, openib.org. 

1) OpenIB represents a number of companies coming together with lots of
InfiniBand source code, 
with duplicate code for the access layer and most of the ULPs
2) the SourceForge work is already part of this
3) the foundation of infiniband support will be the Access Layer, so it
needs the community's feedback first
4) we are looking for feedback on both the access layer code in the
current openib snapshot and the access layer code that we submitted a
few weeks ago 
to learn which is more acceptable to the community.

Now to answer a couple specific questions. 

>Hm, without open source drivers, the Intel stack doesn't seem very 
>viable, correct?

Correct, that is why we hope that Mellanox contributes their driver for
IBAL to open source. 

>> The comments you have given on IBAL would probably only take a few
>> weeks to change.
>Is that work already underway?  Finished?  If neither, why not?

Work is, or at least was underway, but
we put it aside last week to review the rest of the code now in
openib.org.
We also need an open source driver.

>What are the issues with the OpenIB stack?  

As I stated above, we are part of the openib.org collaboration and 
will be working on helping develop a stack that is "best of
breed" from all of the available code. Starting from the bottom up, 
we first need to review the various proposals for the 
Access Layer and determine which code base to start with. 
The initial agreement was to use the 
TopSpin code for an access layer. This agreement was made before anyone
got to see any code. 
After review of this code, we think it needs a lot of work. We were
waiting for the openib.org email lists to open and sending in comments
there.
That way we could work a lot of details offline from lkml, since 
lots of discussion will be needed. 

But since you asked here are a few,

1.) The tsapi APIs look like Windows APIs (at least in the original
drop)
2.) Looking at the API specification document, 
It is missing lots of verbs required by the InfiniBand Specification
CloseCA, ModifyAV, QueryCQ, CreateEEC, ModifyEEC, QueryEEC, 
DestroyEEC, QueryMR, ReregisterMR, ReregisterPhysMR, RegisterSharedMR, 
AllocMW, QueryMW, BindMW, and FreeMW
3.) The code is not compliant with the InfiniBand specification and has
proprietary
implementations of things like "path records" so it will only work with
the 
TopSpin subnet manager that requires you to buy a topspin switch.
4.) Not sure if they have fixed this yet in the 2.6 code, but the 2.4
code 
has like 18 different loadable modules. This could probably be collapsed
into 5, one for the HCA driver, one for the access layer, one for the
IPoIB driver, one for the SRP driver and one for the SDP driver. 
5.) There is no user-mode access layer requiring ULPs to code to the HCA

user-mode driver APIs directly. 
This will mean that new user mode ULPs will need to be 
developed for each new HCA that comes along. 
6.)The VAPI code has extra propietary verbs that are not specified by
the InfiniBand 
Specification. 
7.) The implementation is deficient in it's support for InfiniBand
management
services, like the required RMPP protocol, MAD services, SA query helper
functions. 
8.) Some of the message fields of the CM are hard coded. 
9.) The CM does not support reliable datagrams. 
10) There is no built in support for plug and play events, port up/down,
LID change, SM change,
11) VAPI call stack is deep and puts a lot of big data structure on the
stack. 

There is more, but as I stated before, we suggest discussing most of
these issues within
openib.org first, trying to come to agreement on what is best and then
review our 
suggestions with lkml to make sure we are one track. 

>If there are any, how does the Intel stack solve those 
>issues?  

The SourceForge code IBAL(not just developed by Intel but has
contributions from several companies,
including InfiniCon, Mellanox, Fujitsu and Intel) 
is feature complete and compliant with the InfiniBand specification. It
may not be quite as
hardened as the TopSpin stack, but that gap is rapidly closing. 
We'd also like to know from the other openib.org people, 
What are the issues with the SourceForge IBAL ? 

We know the issues raised by lkml and think these can be fixed. 

The biggest problem I see is that we do not have an open source HCA
driver
and that could be fixed too, if Mellanox wanted to, or someone could
take the VAPI code they open sourced and port it to IBAL. 

>Could the Intel solutions be merged 
>into the OpenIB stack to solve these issues?

Given there are so many issues with the TSAPI, would it be easier to 
fix the issues lkml raised with IBAL and port the "best of breed" ULPs
to it ?
Since all the tsAPIs will have to change anyway, to non-Windows-ize
them, 
all the ULPs will need to be re-ported again anyway. 

Re: PATCH - InfiniBand Access Layer (IBAL)

[Greg KH]

On Sat, Mar 13, 2004 at 07:46:12PM -0800, Woodruff, Robert J wrote:
> On Sat, Mar 13, 2004 at 02:07:13PM -0800, Greg KH wrote:
> >I think you need to work with the openib.org people as they seem to
> >have:
> >	- working code with support for a number of different devices
> I have code that has been running MPI NBPs, Pallas benchmark, and IPoIB
> stress tests for almost 10 days now on multiple vendors equipment. We
> are also close to having a major data base vendor's DAPL stress test
> running very well.  I'm thinkin this code could be running 1000 nodes
> clusters within a couple of months, if people wanted to try it, but
> that would require Mellanox to open source their TVPD. 

Hm, without open source drivers, the Intel stack doesn't seem very
viable, correct?

> >	- developers with extensive kernel programming experience
> As if we don't. 

Former kernel subsystem maintainers?  I did not realize that.

> >	  working on cleaning up the code to fit properly into the
> >	  kernel tree.
> The comments you have given on IBAL would probably only take a few weeks
> to change.

Is that work already underway?  Finished?  If neither, why not?

> >	- their code showing up in at least one distro which will expose
> >	  them to a much wider range of testing than Intel's project so
> >	  far has had.
> Ok, the OEMS pushed for and got a distro to accept a TTM solution,
> that's OK for getting InfiniBand jumpstarted, but does that mean we
> have to accept it into Linux and live with that solution forever. 

We constantly change the kernel.  Look at the USB stack.  It has been
rewritten completely almost 3 times now.  Did anyone really notice
besides the wonderful speed improvements?  :)

We never have to "live with a solution forever."

> I guess I just wish they had started working with us on this open source
> project 2 years ago when we started it, rather than developing a
> complete stack behind closed doors and then releasing it without any
> input from anyone.

There have been numerous closed source IB stacks for Linux.  Just like
there were numerous Bluetooth stacks for Linux in the past.  Over time
the closed ones are weeded out as everyone relies on the in-kernel one.

> Now there are serious issues that will take a lot of work to fix.

What are the issues with the OpenIB stack?  If there are any, how does
the Intel stack solve those issues?  Could the Intel solutions be merged
into the OpenIB stack to solve these issues?

> At least our project was totally open from the start and anyone could
> have provided input at any time. 

I understand your frustrations, I've been there before.

Good luck,

greg k-h

RE: PATCH - InfiniBand Access Layer (IBAL)

[Woodruff, Robert J]

On Sat, Mar 13, 2004 at 02:07:13PM -0800, Greg KH wrote:
>I think you need to work with the openib.org people as they seem to
>have:
>	- working code with support for a number of different devices
I have code that has been running MPI NBPs, Pallas benchmark, and IPoIB
stress tests
for almost 10 days now on multiple vendors equipment. We are also close
to having
a major data base vendor's DAPL stress test running very well.  I'm
thinkin this code could be
running 1000 nodes clusters within a couple of months, if people wanted
to try it,
but that would require Mellanox to open source their TVPD. 
>	- developers with extensive kernel programming experience
As if we don't. 
>	  working on cleaning up the code to fit properly into the
>	  kernel tree.
The comments you have given on IBAL would probably only take a few weeks
to change.
>	- their code showing up in at least one distro which will expose
>	  them to a much wider range of testing than Intel's project so
>	  far has had.
Ok, the OEMS pushed for and got a distro to accept a TTM solution,
that's OK for getting
InfiniBand jumpstarted, 
but does that mean we have to accept it into Linux 
and live with that solution forever. 

I guess I just wish they had started working with us on this open source
project 2 years ago
when we started it, rather than developing a complete stack behind
closed doors and then
releasing it without any input from anyone. Now there are serious issues
that will take a lot 
of work to fix. At least our project was totally open from the 
start and anyone could have provided input at any time. 

Enough said.

I understand your position.  We will submit our code to openib.org. 

Re: PATCH - InfiniBand Access Layer (IBAL)

[Greg KH]

On Sat, Mar 13, 2004 at 02:07:13PM -0800, Woodruff, Robert J wrote:
> 
> The Mellanox and Topspin folks along with some people from some of
> the national labs are trying to start up a website called openib.org,
> with data bases, email lists, etc. where people can submit code for
> this "best of breed" stack and discuss it. As long as it is truly 
> open, the linux community is allowed to participate, and the code
> is evaluated on it's technical merits, rather than one companies
> personal agenda, this forum might serve as a way for us to sort
> through this without taking every issue to lkml. 
> 
> What are your thoughts on how we should proceed ? 

I think you need to work with the openib.org people as they seem to
have:
	- working code with support for a number of different devices
	- developers with extensive kernel programming experience
	  working on cleaning up the code to fit properly into the
	  kernel tree.
	- their code showing up in at least one distro which will expose
	  them to a much wider range of testing than Intel's project so
	  far has had.

So it seems that you really need to work with them if you wish to get
your code merged into theirs, as theirs already seems to be an almost
complete solution...

Good luck,

greg k-h

Re: PATCH - InfiniBand Access Layer (IBAL)

[Andrew Morton]

"Woodruff, Robert J" <woody@co.intel.com> wrote:
>
> The Mellanox and Topspin folks along with some people from some of
>  the national labs are trying to start up a website called openib.org,
>  with data bases, email lists, etc. where people can submit code for
>  this "best of breed" stack and discuss it. As long as it is truly 
>  open, the linux community is allowed to participate, and the code
>  is evaluated on it's technical merits, rather than one companies
>  personal
>  agenda, this forum might serve as a way for us to sort through this
>  without
>  taking every issue to lkml. 

That is, of course, an excellent approach.

But beware of being *too* disconnected from the lists@vger.kernel.org.  We
don't want to get in the situation where you pop up with a couple of
person-years' worth of work and other kernel developers have major issues
with it.  Please find a balance - some way of regularly checkpointing.

Also, beware of aiming for a finished product.  We would *much* prefer that
a simple, minimal core, maybe with just a single device driver be merged
into the mainline kernel.  The IBAL developers then proceed to enhance
that, sending regular updates.  Every couple of weeks would suit.

That way everyone else can see the code evolving, and can help, and can
understand.  And other people will fix your bugs for you, and update your
code as kernel-wide changes are implemented.  And we all avoid nasty
surprises and extensive rework.

RE: PATCH - InfiniBand Access Layer (IBAL)

[Woodruff, Robert J]

On Wed, 25 Feb 2004, Linus Torvalds wrote:

>Oh, I agree that _reviewing_ code is good, together with feedback 
>on what would improve its chances of getting accepted later on. 
>But it should be clear that regardless, 
>we don't add features that nobody 
>can sanely test and where hardware isn't available.
>
>		Linus

Just wanted to give an update on where we are with the InfiniBand IBAL.

After we submitted the IBAL code for review, there has been additional
InfiniBand code put into open source by the major InfiniBand companies,
namely TopSpin, Mellanox, InfiniCon, and Voltaire.
This presents somewhat of a problem because now there are multiple 
versions of the various InfiniBand S/W components, Access Layers, and
ULPs.

We now need to review all this code and
determine which code is best to use as a basis for something that 
we would eventually try to get into Linux. We hope that people in the
linux community will help us sort through this mess and come up with
a "best of breed" stack out of all the available code. 

The Mellanox and Topspin folks along with some people from some of
the national labs are trying to start up a website called openib.org,
with data bases, email lists, etc. where people can submit code for
this "best of breed" stack and discuss it. As long as it is truly 
open, the linux community is allowed to participate, and the code
is evaluated on it's technical merits, rather than one companies
personal
agenda, this forum might serve as a way for us to sort through this
without
taking every issue to lkml. 

What are your thoughts on how we should proceed ? 

Re: PATCH - InfiniBand Access Layer (IBAL)

[Sam Ravnborg]

On Wed, Feb 25, 2004 at 10:05:32AM -0800, Linus Torvalds wrote:
> 
> 
> On Wed, 25 Feb 2004, Sam Ravnborg wrote:
> > 
> > If we take the vendor persåective here. Then why should they make their
> > driver open source, when the middle layer is not part of the 
> > main stream kernel?
> 
> And why should we take the vendor perspective?

The people developing Inifiband support ask for a review.
And to me it looks too easy to say "no open source driver", so 
do not expect a review. Everything is voluntary of course.

Inclusion in the kernel is for me a different matter.

> We don't add drivers for stuff that doesn't exist, and is not even likely 
> to be used much. That way, when problems occur (and they _will_ occur), 
> the burden of the crap will be firmly on the shoulders of the people who 
> should care.

Yep, agree on that - but does not conflict when what I meant to say.

	Sam

Re: PATCH - InfiniBand Access Layer (IBAL)

[Timothy Miller]

Linus Torvalds wrote:
> 
> On Wed, 25 Feb 2004, Sam Ravnborg wrote:
> 
>>If we take the vendor persåective here. Then why should they make their
>>driver open source, when the middle layer is not part of the 
>>main stream kernel?
> 
> 
> And why should we take the vendor perspective?
> 
> We don't add drivers for stuff that doesn't exist, and is not even likely 
> to be used much. That way, when problems occur (and they _will_ occur), 
> the burden of the crap will be firmly on the shoulders of the people who 
> should care.

I was not suggesting that we add drivers or even the middle layer to the 
mainline kernel until there is some use for it.  Rather, it wouldn't be 
a bad idea for someone to DEVELOP it as a patch which could be mainlined 
the moment there became a demand for it.  This also leaves the work in 
the hands of those who should care, because anyone using the middle 
layer has the 'disadvantage' of using a non-standard patch which they 
have to take responsibility for using.

Re: PATCH - InfiniBand Access Layer (IBAL)

[Linus Torvalds]

On Wed, 25 Feb 2004, Sam Ravnborg wrote:
> > 
> > And why should we take the vendor perspective?
> 
> The people developing Inifiband support ask for a review.

Oh, I agree that _reviewing_ code is good, together with feedback on what
would improve its chances of getting accepted later on. But it should be
clear that regardless, we don't add features that nobody can sanely test
and where hardware isn't available.

		Linus

Re: PATCH - InfiniBand Access Layer (IBAL)

[Sam Ravnborg]

> >
> >I'm sure infinibad will be inetresting once htere are
> >actual hardware driver.s  However, I'm not aware of any
> >open source drivers in existnace now, so what good is
> >a stack ?
> >
> 
> Chicken and egg.  If infiniband has some significant value, it would be 
> in everyone's favor if we took the initiative.

If we take the vendor persåective here. Then why should they make their
driver open source, when the middle layer is not part of the 
main stream kernel?

So short-term I beleive the
"no open source drivers => no interest in IBAL"
is a bit to fast a conclusion.

Let's anyway see what they have, and give them feedback.
Based on the comments I have seen so far there is
a good chance to see open source drivers before IBAL is
considered ready for main stream kernel.

	Sam

Re: PATCH - InfiniBand Access Layer (IBAL)

[Linus Torvalds]

On Wed, 25 Feb 2004, Sam Ravnborg wrote:
> 
> If we take the vendor persåective here. Then why should they make their
> driver open source, when the middle layer is not part of the 
> main stream kernel?

And why should we take the vendor perspective?

We don't add drivers for stuff that doesn't exist, and is not even likely 
to be used much. That way, when problems occur (and they _will_ occur), 
the burden of the crap will be firmly on the shoulders of the people who 
should care.

			Linus

Re: PATCH - InfiniBand Access Layer (IBAL)

[Roland Dreier]

    Timothy> On the other hand, if something else is better or
    Timothy> adequate, like PCI Express (wasn't that based in
    Timothy> infiniband?), then there's no point.

PCI Express is not an InfiniBand replacement.  While it is true (I
think this is what you meant) that the PCI Express electrical
signaling is based on InfiniBand (they both use multiple lanes of 2.5
Gb/sec high-speed serial), the upper layers of the two standards are
quite different.  PCI Express and InfiniBand are really quite
complementary.  In fact (just to plug my employer :) we have
demonstrated an Infiniband host adapter that has two 4X IB (10 Gb/sec
ports) and plugs into an 8X PCI Express slot:
  http://www.topspin.com/news/pressrelease/pr_021704b.html

PCI Express (once products ship) will be essentially a faster PCI-X
replacement.  You will get a system with PCI Express slots and plug
PCI Express adapter cards into them.  There is something called PCI
Express "Advanced Switching" but that is quite a long way away from
being something you could build a cluster with.

InfiniBand on the other hand has evolved into a cluster interconnect.
In the beginning it was pitched as a PCI replacement but that was
given up long ago.  However, it evolved into an excellent cluster
interconnect.  Right now you can buy 10 Gb/sec host adapters and a
variety of switches (up to 96 ports).  There are a number of quite
large IB clusters already built and in production already.

Best,
  Roland

Re: PATCH - InfiniBand Access Layer (IBAL)

[Timothy Miller]

Rik van Riel wrote:
> On Wed, 25 Feb 2004, Matti Aarnio wrote:
> 
> 
>>People building "cheap supercomputers" will be going that way
>>most definitely.  Slowest version is 2.5 Gbit/s, and most
>>common one appears to be running 4x that.
> 
> 
> I'm sure infinibad will be inetresting once htere are
> actual hardware driver.s  However, I'm not aware of any
> open source drivers in existnace now, so what good is
> a stack ?
> 

Chicken and egg.  If infiniband has some significant value, it would be 
in everyone's favor if we took the initiative.  On the other hand, if 
something else is better or adequate, like PCI Express (wasn't that 
based in infiniband?), then there's no point.

Re: PATCH - InfiniBand Access Layer (IBAL)

[Christoph Hellwig]

On Wed, Feb 25, 2004 at 02:28:19AM +0200, Matti Aarnio wrote:
> Another thing (for which I would have use at hand right away, actually)
> is to have fully functional Fibre Channel subsystem in Linux along
> with drivers to modern cards e.g. JNI's.  (2 Gbit/s FC)
> 
> Plugging tens of terabytes of disks on a box is somewhat challenging
> without resorting to that technology...

There's better troll than you, Matti :)

In a stock 2.6 kernel you get support for the following 2GB FC adapters:

  Qlogic 2xxx/6xxx (PCI-X/ PCI-Express)
  LSI Fusion
  IBM zfcp (okay, you need a mainframe for that, but.. :))

if you like a crappy out of tree vendor driver you can also get Emulex
support.  You belowed vendor JNI unfortunately only has binary only drivers
for intel plattforms and 2.2/2.4 kernels.

Re: PATCH - InfiniBand Access Layer (IBAL)

[Rik van Riel]

On Wed, 25 Feb 2004, Matti Aarnio wrote:

> People building "cheap supercomputers" will be going that way
> most definitely.  Slowest version is 2.5 Gbit/s, and most
> common one appears to be running 4x that.

I'm sure infinibad will be inetresting once htere are
actual hardware driver.s  However, I'm not aware of any
open source drivers in existnace now, so what good is
a stack ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

RE: PATCH - InfiniBand Access Layer (IBAL)

[Rik van Riel]

On Tue, 24 Feb 2004, Woodruff, Robert J wrote:

> You may want to start talking more with some of your customers. I'm
> thinkin they might have a different opinion about wanting InfiniBand
> support in Linux.

What would customers do with an infiniband layer for
which no device drivers are available ?

Now if the device drivers were available, it would
make sense...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: PATCH - InfiniBand Access Layer (IBAL)

[Matti Aarnio]

On Tue, Feb 24, 2004 at 05:29:16PM -0500, Rik van Riel wrote:
> On Tue, 24 Feb 2004, Greg KH wrote:
> > On Tue, Feb 24, 2004 at 07:50:18PM +0000, Christoph Hellwig wrote:
> > > I don't understand why anyone is wasting time on this.  Without available
> > > hardware drivers this huge midlayer is completely useless.
> > You mean this whole huge chunk of code doesn't have any hardware
> > drivers?  What good is it then?
> 
> Beats me. I hope we can just bury this infiniband stuff
> before we waste any more time on it.
> 
> I really can't see any reason why we would want to have
> this.

Maybe you don't, but some do.  Uses are rare, of course, but:
  http://www.apple.com/education/science/profiles/vatech/

People building "cheap supercomputers" will be going that way
most definitely.  Slowest version is 2.5 Gbit/s, and most
common one appears to be running 4x that.


Another thing (for which I would have use at hand right away, actually)
is to have fully functional Fibre Channel subsystem in Linux along
with drivers to modern cards e.g. JNI's.  (2 Gbit/s FC)

Plugging tens of terabytes of disks on a box is somewhat challenging
without resorting to that technology...

I just don't have luxury of having a year or two to spend on
development of necessary things, I need to choose systems with
the support readily in place so I can load in my applications,
and start using them.  :-/


/Matti Aarnio

RE: PATCH - InfiniBand Access Layer (IBAL)

[Woodruff, Robert J]

On Tue, 24 Feb 2004,  Rik van Riel wrote:
>Beats me. I hope we can just bury this infiniband stuff
>before we waste any more time on it.

>I really can't see any reason why we would want to have
>this.

You may want to start talking more with some of your customers. I'm
thinkin
they might have a different opinion about wanting InfiniBand support in
Linux. 
You can hide your head in the sand and hope it goes away, but I don't
think
it is going to. 

Re: PATCH - InfiniBand Access Layer (IBAL)

[Rik van Riel]

On Tue, 24 Feb 2004, Greg KH wrote:
> On Tue, Feb 24, 2004 at 07:50:18PM +0000, Christoph Hellwig wrote:

> > I don't understand why anyone is wasting time on this.  Without available
> > hardware drivers this huge midlayer is completely useless.
> 
> You mean this whole huge chunk of code doesn't have any hardware
> drivers?  What good is it then?

Beats me. I hope we can just bury this infiniband stuff
before we waste any more time on it.

I really can't see any reason why we would want to have
this.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

RE: PATCH - InfiniBand Access Layer (IBAL)

[Woodruff, Robert J]

On Tue, Feb 24, 2004 at 07:50:18PM +0000, Greg KH wrote:

>Please make those changes and then post the patch here 
>(not just a link, if it's too big, split it up into the logical pieces
to fit.)  
> We can go from there.

> You mean this whole huge chunk of code doesn't have any hardware
drivers?  
> What good is it then?

Good to split things into chunks so that it is more manageable for
review
by the Linux community. 

Hardware drivers coming soon.
I saw an email on the InfiniBand sourceforge list 
a couple weeks back that the Mellanox people will be providing open
source code 
for their InfiniBand hardware pretty soon. It may need some rework to
meet the 
standards of the Linux community and fit into the reworked IBAL. I also
saw 
a note from a guy from Fujitsu on the Infiniband list 
that they have a 2.6 hardware driver for IBAL for their HCA, but I am
not 
sure of their open source plans. 
Pretty soon, I don't think people will need worry about a lack of
InfiniBand open 
source code. There may be more of it than you will want to deal with. 

BTW. The patch on sourceforge should be OK now. My error on initial
upload
and now fixed. 

RE: PATCH - InfiniBand Access Layer (IBAL)

[Woodruff, Robert J]

On Tue, Feb 24, 2004 at 11:45:04AM -0800, Greg KH  wrote:
>The
http://osdn.dl.sourceforge.net/sourceforge/infiniband/patch-2_6_3-iba.bz
2
>patch is corrupted.
 
I may have made a mistake in upload to sourceforge. 
I tried to upload both a compressed patch file and a compressed 
tar file for those that do not want to deal with patches. 
I think it should be Ok now.  
Jerrie could you download the patch from sourceforge and take a 
look at it to make sure it is OK. 

>Please make those changes and then post the patch here 
>(not just a link, if it's too big, split it up into 
>the logical pieces to fit.)  We can go from there.

Ok, we will do that too. 

Re: PATCH - InfiniBand Access Layer (IBAL)

[Greg KH]

On Tue, Feb 24, 2004 at 07:50:18PM +0000, Christoph Hellwig wrote:
> On Tue, Feb 24, 2004 at 11:44:30AM -0800, Greg KH wrote:
> > Please make those changes and then post the patch here (not just a link,
> > if it's too big, split it up into the logical pieces to fit.)  We can go
> > from there.
> 
> I don't understand why anyone is wasting time on this.  Without available
> hardware drivers this huge midlayer is completely useless.

You mean this whole huge chunk of code doesn't have any hardware
drivers?  What good is it then?

greg k-h

Re: PATCH - InfiniBand Access Layer (IBAL)

[Christoph Hellwig]

On Tue, Feb 24, 2004 at 11:44:30AM -0800, Greg KH wrote:
> Please make those changes and then post the patch here (not just a link,
> if it's too big, split it up into the logical pieces to fit.)  We can go
> from there.

I don't understand why anyone is wasting time on this.  Without available
hardware drivers this huge midlayer is completely useless.

Re: PATCH - InfiniBand Access Layer (IBAL)

[Greg KH]

On Tue, Feb 24, 2004 at 11:29:04AM -0800, Woodruff, Robert J wrote:
> 
> Since it is rather big,
> I have posted a patch for the InfiniBand Access Layer (IBAL) 
> for the linux-2.6.3 kernel on sourceforge in the file releases
> area. 
> http://sourceforge.net/projects/infiniband

The
http://osdn.dl.sourceforge.net/sourceforge/infiniband/patch-2_6_3-iba.bz2
patch is corrupted.

> We have already received some comments from Greg and others(such as 
> not liking the Component library (Complib) abstraction layer)
> and we are starting to make those changes. Greg suggested that
> we submit the code to the larger Linux community for review, 
> so here it is. 

Please make those changes and then post the patch here (not just a link,
if it's too big, split it up into the logical pieces to fit.)  We can go
from there.

thanks,

greg k-h

PATCH - InfiniBand Access Layer (IBAL)

[Woodruff, Robert J]

Since it is rather big,
I have posted a patch for the InfiniBand Access Layer (IBAL) 
for the linux-2.6.3 kernel on sourceforge in the file releases
area. 
http://sourceforge.net/projects/infiniband

We have already received some comments from Greg and others(such as 
not liking the Component library (Complib) abstraction layer)
and we are starting to make those changes. Greg suggested that
we submit the code to the larger Linux community for review, 
so here it is. 

Please let us know what other changes might be needed so that it is
acceptable for inclusion into the 2.6 base.