rdma-core 12 breaks libibverbs ABI

rdma-core 12 breaks libibverbs ABI

[Benjamin Drung]

Hi,

the rmda-core commit 683e55aa5d291c8fda338dae565fbdae0c0000b5 "ibverbs:
Allow vendor data response in create_ah command" breaks the ABI of the
libibverbs library: The symbol ibv_cmd_create_ah@IBVERBS_1.0 is replaced
by ibv_cmd_create_ah@IBVERBS_1.4. Please bring back ibv_cmd_create_ah@IB
VERBS_1.0 to restore the ABI. Otherwise the version of the soname needs
to be bumped from 1 to 2.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL:  http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Andreas Gauger, Achim Weiss.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Jason Gunthorpe]

On Thu, Jan 26, 2017 at 04:55:47PM +0100, Benjamin Drung wrote:
> the rmda-core commit 683e55aa5d291c8fda338dae565fbdae0c0000b5 "ibverbs:
> Allow vendor data response in create_ah command" breaks the ABI of the
> libibverbs library: The symbol ibv_cmd_create_ah@IBVERBS_1.0 is replaced
> by ibv_cmd_create_ah@IBVERBS_1.4. Please bring back ibv_cmd_create_ah@IB
> VERBS_1.0 to restore the ABI. Otherwise the version of the soname needs
> to be bumped from 1 to 2.

The ibv_cmd_* family of symbols are now considered private internal
symbols only for use between libibverbs and the included plugin libraries.

The decision was made to stop supporting providers outside of
rdma-core, so the library SONAME only reflects the public facing ABI.

I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch
was merged, but before we release 13 we need to drop all the ibv_cmd_*
entry points from the public headers so that nothing outside rdma-core
can link to these symbols.

AFAIK this is an accepted way to build a library that works with
private loadable plugins.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Christoph Hellwig]

On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote:
> I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch
> was merged, but before we release 13 we need to drop all the ibv_cmd_*
> entry points from the public headers so that nothing outside rdma-core
> can link to these symbols.

A good thing would be to move all these symbols into a @IBVERBS_PRIVATE
namespace, similar to what glibc does for it's internal library only
exports.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Jason Gunthorpe]

On Thu, Jan 26, 2017 at 09:35:48AM -0800, Christoph Hellwig wrote:
> On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote:
> > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch
> > was merged, but before we release 13 we need to drop all the ibv_cmd_*
> > entry points from the public headers so that nothing outside rdma-core
> > can link to these symbols.
> 
> A good thing would be to move all these symbols into a @IBVERBS_PRIVATE
> namespace, similar to what glibc does for it's internal library only
> exports.

Makes sense for clarity.

I guess we should remove the public headers and change the symbol
version all at once in '13'? This would instantly make all out of tree
providers totally unworkable.

Or go back to providing strong compat to the providers as Ben
suggested..  In this case someone will have to vote for this with a
patch to put the missing symbol back...

I know mlx voted for the first path - Ira?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Christoph Hellwig]

On Thu, Jan 26, 2017 at 11:09:26AM -0700, Jason Gunthorpe wrote:
> Or go back to providing strong compat to the providers as Ben
> suggested..  In this case someone will have to vote for this with a
> patch to put the missing symbol back...

There should not be any compat - it's just going to drag down
development and everyone is free to submit their provider to rdma-core,
same as kernel drivers.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Jason Gunthorpe]

On Thu, Jan 26, 2017 at 10:11:13AM -0800, Christoph Hellwig wrote:
> On Thu, Jan 26, 2017 at 11:09:26AM -0700, Jason Gunthorpe wrote:
> > Or go back to providing strong compat to the providers as Ben
> > suggested..  In this case someone will have to vote for this with a
> > patch to put the missing symbol back...
> 
> There should not be any compat - it's just going to drag down
> development and everyone is free to submit their provider to rdma-core,
> same as kernel drivers.

Yep, that was the argument that won the day so far.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On Thu, Jan 26, 2017 at 10:11:13AM -0800, Christoph Hellwig wrote:
> On Thu, Jan 26, 2017 at 11:09:26AM -0700, Jason Gunthorpe wrote:
> > Or go back to providing strong compat to the providers as Ben
> > suggested..  In this case someone will have to vote for this with a
> > patch to put the missing symbol back...
>
> There should not be any compat - it's just going to drag down
> development and everyone is free to submit their provider to rdma-core,
> same as kernel drivers.

Agree, I have no desire to support out-of-tree providers.

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: rdma-core 12 breaks libibverbs ABI

[Hefty, Sean]

> > There should not be any compat - it's just going to drag down
> > development and everyone is free to submit their provider to rdma-
> core,
> > same as kernel drivers.
> 
> Yep, that was the argument that won the day so far.

I don't disagree with the direction, but I agree with Benjamin.  The intent behind a public symbol/structure and their actual usage by applications may differ.  Why wasn't the soname updated?

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Jason Gunthorpe]

On Thu, Jan 26, 2017 at 08:38:40PM +0000, Hefty, Sean wrote:
> > > There should not be any compat - it's just going to drag down
> > > development and everyone is free to submit their provider to rdma-
> > core,
> > > same as kernel drivers.
> > 
> > Yep, that was the argument that won the day so far.
> 
> I don't disagree with the direction, but I agree with Benjamin.  The
> intent behind a public symbol/structure and their actual usage by
> applications may differ.  Why wasn't the soname updated?

Well, no it cannot differ. If apps use private symbols they get
burned. Glibc and gcc have established that precedent decades ago.

The soname is not bumped because libibverbs is a symver based library,
which means we can manage our private ABI safely without ever
requiring soname changes. If something does manage to link to it then
it gets a runtime dynamic linking failure, not a random crash.

Remember, libibverbs fundamentally does not support two copies of the
library in the same address space, so one of the major motivations to
use a soname to manage version is invalid. Even worse, because of
this, if we did bump the soname, it would be very painful and
dangerous.

So that leaves us with only one point to argue on: Should libibverbs
have a private ABI or not? If yes, then we are doing it properly.

The issue is muddied a bit because the private ABI has not been as
absolutely private as I'd like, so we could add compat for the two
symbols and defer the incompat transition till a later release.. This
argument would be best supported if someone knows of something that
actually breaks out in the wild...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core 12 breaks libibverbs ABI

[Hefty, Sean]

> > There should not be any compat - it's just going to drag down
> > development and everyone is free to submit their provider to rdma-
> core,
> > same as kernel drivers.
> 
> Agree, I have no desire to support out-of-tree providers.

What is the plan for handling live updates to a single vendor library?

Is the intent that vendors must now (potentially) provide different versions of their libraries, based on which libibverbs release is deployed by the user?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Jason Gunthorpe]

On Thu, Jan 26, 2017 at 11:53:36PM +0000, Hefty, Sean wrote:
> > > There should not be any compat - it's just going to drag down
> > > development and everyone is free to submit their provider to rdma-
> > core,
> > > same as kernel drivers.
> > 
> > Agree, I have no desire to support out-of-tree providers.
> 
> What is the plan for handling live updates to a single vendor library?
>
> Is the intent that vendors must now (potentially) provide different
> versions of their libraries, based on which libibverbs release is
> deployed by the user?

Well, this has been the case for awhile, at least for certain
providers (eg mlx5 will not link to older libibverbs).

The patch to add IBVERBS_PRIVATE_xx certainly makes it every
provider's problem..

As I see it, anyone doing an update driver distribution will have to
QA it with a certain libibverbs, so I doubt there is much point in
making it too easy for a user to mix and match too widely??

So vendors have a few basic choices:
- Ship libibverbs.rpm with the updated driver and matched libibverbs
  This is pretty easy, I probably would recommend this.
- QA the driver with the libibverbs in the various targetted distro
  releases, as you suggested
- Ship their fix to the distro and have the distro push the update

It isn't really any different from the problem a vendor would face
getting the kernel driver updated..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core 12 breaks libibverbs ABI

[Hefty, Sean]

> - Ship libibverbs.rpm with the updated driver and matched libibverbs
>   This is pretty easy, I probably would recommend this.

This has issues if there are multiple providers present.  Maybe that doesn't happen in non-testing environments.  I wonder if the distros would agree with this approach.

> - QA the driver with the libibverbs in the various targetted distro
>   releases, as you suggested
> - Ship their fix to the distro and have the distro push the update
> 
> It isn't really any different from the problem a vendor would face
> getting the kernel driver updated..

I agree.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 1116 bytes --]

On 1/26/2017 8:17 PM, Hefty, Sean wrote:
>> - Ship libibverbs.rpm with the updated driver and matched libibverbs
>>   This is pretty easy, I probably would recommend this.
> 
> This has issues if there are multiple providers present.  Maybe that doesn't happen in non-testing environments.  I wonder if the distros would agree with this approach.

I don't know how much I would rely on the belief that mixed environments
don't happen outside of testing.  In particular, the structure of Lustre
filesystems or the need to gateway from a fabric that one brand of
storage server doesn't support to another one would both seem to be
times when mixed hardware might exist.  For that matter, the typical
stock trading RDMA environment mixes IB and Ethernet, maybe in the
future it will have OPA in there too?  I think you are right that it is
the most common, but I don't think we can rely on it being that way all
the time.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG Key ID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: rdma-core 12 breaks libibverbs ABI

[Jason Gunthorpe]

On Fri, Jan 27, 2017 at 09:32:51AM -0500, Doug Ledford wrote:
> On 1/26/2017 8:17 PM, Hefty, Sean wrote:
> >> - Ship libibverbs.rpm with the updated driver and matched libibverbs
> >>   This is pretty easy, I probably would recommend this.
> > 
> > This has issues if there are multiple providers present.  Maybe
> > that doesn't happen in non-testing environments.  I wonder if the
> > distros would agree with this approach.
> 
> I don't know how much I would rely on the belief that mixed environments
> don't happen outside of testing.

Yes, I've heard many complaints in production in this area.

As it stands today the changes to libibverbs in vendor trees are too
extensive, the only way to resolve them is to merge the vendors
patches into one source tree and build a custom release. I think
rdma-core will make that much simpler.

Making the ABI private or public doesn't really help things either
way, as far as I can tell. Remember the stable ABI is upstream, not
in a vendor tree, so don't expect a random vendor libibverbs to work
with any other providers, even today! They might link, but based on
what I've seen they will be subtly broken.

Ideally vendors will get their stuff upstream and a site could choose
to just use upstream rdma-core instead of a vendor tree.

I would still suggest vendors shipping libibverbs.rpm as their update
is and reasonable and easy approach for their simple users.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core 12 breaks libibverbs ABI

[Weiny, Ira]

> 
> It isn't really any different from the problem a vendor would face getting the
> kernel driver updated..
> 

My concern is that OFA (and the vendors) have not exactly had a good track record of kernel updates in the past.  So I'm not sure that this statement give me a lot of comfort.

That said, I know that most vendors seem to be making a good effort at integrating upstream.  And there is still a mechanism which allows for an external provider.

So I'm not opposed to this.

Ira

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Benjamin Drung]

Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig:
> On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote:
> > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4
> > patch
> > was merged, but before we release 13 we need to drop all the
> > ibv_cmd_*
> > entry points from the public headers so that nothing outside rdma-
> > core
> > can link to these symbols.
> 
> A good thing would be to move all these symbols into a
> @IBVERBS_PRIVATE
> namespace, similar to what glibc does for it's internal library only
> exports.

I am fine with moving the private symbols into a IBVERBS_PRIVATE_XX
namespace as long as all consumers of the private symbols are build
from the rdma-core source package. I need to check if there is a
package in Debian that uses the private symbols.

So either go forward (extra namespace for private symbols) or revert
the ABI breakage (for the time being). The discussion on this list seem
to prefer the first option.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL:  http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Andreas Gauger, Achim Weiss.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Benjamin Drung]

Am Dienstag, den 31.01.2017, 13:23 +0100 schrieb Benjamin Drung:
> Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig:
> > On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote:
> > > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4
> > > patch
> > > was merged, but before we release 13 we need to drop all the
> > > ibv_cmd_*
> > > entry points from the public headers so that nothing outside
> > > rdma-
> > > core
> > > can link to these symbols.
> > 
> > A good thing would be to move all these symbols into a
> > @IBVERBS_PRIVATE
> > namespace, similar to what glibc does for it's internal library
> > only
> > exports.
> 
> I am fine with moving the private symbols into a IBVERBS_PRIVATE_XX
> namespace as long as all consumers of the private symbols are build
> from the rdma-core source package. I need to check if there is a
> package in Debian that uses the private symbols.

I brought this question to the debian-devel mailing list:
https://lists.debian.org/debian-devel/2017/01/msg00791.html

One private feedback was that removing symbols without bumping the
soname is a really bad idea (even if they are private symbols). So one
option would be to bump the soname once and remove all private symbols
in that run.

I searched for "ibv_cmd_" in Debian's source code:
https://codesearch.debian.net/search?q=ibv_cmd_
It boils down to these source packages:

shipped by rdma-core
====================

* librdmacm
* libibverbs

provided by ibverbs-providers
=============================

* libcxgb3
* libmlx5
* libmthca
* libnes
* libipathverbs
* libmlx4

remaining
=========

* openmpi

openmpi uses ibv_cmd_open_xrcd in config/opal_check_openfabrics.m4:

   # We have to check functions both exits *and* are declared
   # since some distro ship broken ibverbs devel headers
   # IBV_DEVICE_XRC is common to all OFED versions
   # ibv_create_xrc_rcv_qp was added in OFED 1.3
   # ibv_cmd_open_xrcd (aka XRC Domains) was added in  OFED 3.12

So I recommend to contact openmpi to let them change their configure
check to not check for ibv_cmd_open_xrcd any more.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL:  http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Andreas Gauger, Achim Weiss.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Jason Gunthorpe]

On Tue, Jan 31, 2017 at 01:23:03PM +0100, Benjamin Drung wrote:
> Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig:
> > On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote:
> > > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4
> > > patch
> > > was merged, but before we release 13 we need to drop all the
> > > ibv_cmd_*
> > > entry points from the public headers so that nothing outside rdma-
> > > core
> > > can link to these symbols.
> > 
> > A good thing would be to move all these symbols into a
> > @IBVERBS_PRIVATE
> > namespace, similar to what glibc does for it's internal library only
> > exports.
> 
> I am fine with moving the private symbols into a IBVERBS_PRIVATE_XX
> namespace as long as all consumers of the private symbols are build
> from the rdma-core source package. I need to check if there is a
> package in Debian that uses the private symbols.

> So either go forward (extra namespace for private symbols) or revert
> the ABI breakage (for the time being). The discussion on this list seem
> to prefer the first option.

Yes.

We must implement one of the two options before releasing '13'

I have sent patches on github to do the first..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Jason Gunthorpe]

On Tue, Jan 31, 2017 at 04:06:11PM +0100, Benjamin Drung wrote:

> One private feedback was that removing symbols without bumping the
> soname is a really bad idea (even if they are private symbols). So one
> option would be to bump the soname once and remove all private symbols
> in that run.

Bumping the soname is also a really bad idea - we need to pick which
one is less bad :)

> I searched for "ibv_cmd_" in Debian's source code:
> https://codesearch.debian.net/search?q=ibv_cmd_
> It boils down to these source packages:

Neat. I also looked at the removed headerfiles which will cause FTBS.

dmtcp seems to include an old wrong copy of
libibverbs/examples/devinfo.c - I have no idea why.

usr/include/infiniband/arch.h
 dpdk 
 dmtcp
 fio
 qlvnictools
These seem to be using htonll/etc (replace with glibc's htobe64)

usr/include/infiniband/driver.h
 openmpi
 dmtcp
 libfabric
Most of these are probably unnecessary..

infiniband/marshall.h has none

> openmpi uses ibv_cmd_open_xrcd in config/opal_check_openfabrics.m4:
> 
>    # We have to check functions both exits *and* are declared
>    # since some distro ship broken ibverbs devel headers
>    # IBV_DEVICE_XRC is common to all OFED versions
>    # ibv_create_xrc_rcv_qp was added in OFED 1.3
>    # ibv_cmd_open_xrcd (aka XRC Domains) was added in  OFED 3.12
> 
> So I recommend to contact openmpi to let them change their configure
> check to not check for ibv_cmd_open_xrcd any more.

Yes, they will need to make a few small changes

This is a very small amount of usage and other than the htonll stuff,
none of it looks at all right to me.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Coulter, Susan K]

> On Jan 27, 2017, at 7:32 AM, Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> 
> On 1/26/2017 8:17 PM, Hefty, Sean wrote:
>>> - Ship libibverbs.rpm with the updated driver and matched libibverbs
>>>  This is pretty easy, I probably would recommend this.
>> 
>> This has issues if there are multiple providers present.  Maybe that doesn't happen in non-testing environments.  I wonder if the distros would agree with this approach.
> 
> I don't know how much I would rely on the belief that mixed environments
> don't happen outside of testing.  In particular, the structure of Lustre
> filesystems or the need to gateway from a fabric that one brand of
> storage server doesn't support to another one would both seem to be
> times when mixed hardware might exist.  For that matter, the typical
> stock trading RDMA environment mixes IB and Ethernet, maybe in the
> future it will have OPA in there too?

This is happening now - not in the future.
LANL and other DOE labs already have OPA and IB/mlnx on the same host.


>  I think you are right that it is
> the most common, but I don't think we can rely on it being that way all
> the time.
> 
> -- 
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>    GPG Key ID: B826A3330E572FDD
>    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
> 

==================================================
Susan Coulter / HPC-DES
Network Capability Lead
(505) 667-8425
“Once in a while you get shown the light
    In the strangest of places if you look at it right”  Robert Hunter
==================================================







--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core 12 breaks libibverbs ABI

[Christoph Hellwig]

On Tue, Jan 31, 2017 at 06:37:58PM +0000, Coulter, Susan K wrote:
> This is happening now - not in the future.
> LANL and other DOE labs already have OPA and IB/mlnx on the same host.

And have setups with up to three different providers :)

But the whole point is that the new rdma-core setup gives us a sane
way to handle this:  patch your rdma-core tree with whatever vendor
updates you need, same as the kernel.  The current model where vendors
supply a new libibverbs that has all kinds of incompatible hacks is
simply not sustainable.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html