* rdma-core 12 breaks libibverbs ABI @ 2017-01-26 15:55 Benjamin Drung [not found] ` <1485446147.4194.4.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> 0 siblings, 1 reply; 21+ messages in thread From: Benjamin Drung @ 2017-01-26 15:55 UTC (permalink / raw) To: linux-rdma-u79uwXL29TY76Z2rM5mHXA Hi, the rmda-core commit 683e55aa5d291c8fda338dae565fbdae0c0000b5 "ibverbs: Allow vendor data response in create_ah command" breaks the ABI of the libibverbs library: The symbol ibv_cmd_create_ah@IBVERBS_1.0 is replaced by ibv_cmd_create_ah@IBVERBS_1.4. Please bring back ibv_cmd_create_ah@IB VERBS_1.0 to restore the ABI. Otherwise the version of the soname needs to be bumped from 1 to 2. -- Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Geschäftsführer: Andreas Gauger, Achim Weiss. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1485446147.4194.4.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <1485446147.4194.4.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> @ 2017-01-26 17:13 ` Jason Gunthorpe [not found] ` <20170126171325.GA16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 0 siblings, 1 reply; 21+ messages in thread From: Jason Gunthorpe @ 2017-01-26 17:13 UTC (permalink / raw) To: Benjamin Drung; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA On Thu, Jan 26, 2017 at 04:55:47PM +0100, Benjamin Drung wrote: > the rmda-core commit 683e55aa5d291c8fda338dae565fbdae0c0000b5 "ibverbs: > Allow vendor data response in create_ah command" breaks the ABI of the > libibverbs library: The symbol ibv_cmd_create_ah@IBVERBS_1.0 is replaced > by ibv_cmd_create_ah@IBVERBS_1.4. Please bring back ibv_cmd_create_ah@IB > VERBS_1.0 to restore the ABI. Otherwise the version of the soname needs > to be bumped from 1 to 2. The ibv_cmd_* family of symbols are now considered private internal symbols only for use between libibverbs and the included plugin libraries. The decision was made to stop supporting providers outside of rdma-core, so the library SONAME only reflects the public facing ABI. I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch was merged, but before we release 13 we need to drop all the ibv_cmd_* entry points from the public headers so that nothing outside rdma-core can link to these symbols. AFAIK this is an accepted way to build a library that works with private loadable plugins. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <20170126171325.GA16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <20170126171325.GA16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> @ 2017-01-26 17:35 ` Christoph Hellwig [not found] ` <20170126173548.GA15006-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 0 siblings, 1 reply; 21+ messages in thread From: Christoph Hellwig @ 2017-01-26 17:35 UTC (permalink / raw) To: Jason Gunthorpe; +Cc: Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote: > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch > was merged, but before we release 13 we need to drop all the ibv_cmd_* > entry points from the public headers so that nothing outside rdma-core > can link to these symbols. A good thing would be to move all these symbols into a @IBVERBS_PRIVATE namespace, similar to what glibc does for it's internal library only exports. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <20170126173548.GA15006-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <20170126173548.GA15006-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> @ 2017-01-26 18:09 ` Jason Gunthorpe [not found] ` <20170126180926.GB20330-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-01-31 12:23 ` Benjamin Drung 1 sibling, 1 reply; 21+ messages in thread From: Jason Gunthorpe @ 2017-01-26 18:09 UTC (permalink / raw) To: Christoph Hellwig Cc: Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, ira.weiny On Thu, Jan 26, 2017 at 09:35:48AM -0800, Christoph Hellwig wrote: > On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote: > > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch > > was merged, but before we release 13 we need to drop all the ibv_cmd_* > > entry points from the public headers so that nothing outside rdma-core > > can link to these symbols. > > A good thing would be to move all these symbols into a @IBVERBS_PRIVATE > namespace, similar to what glibc does for it's internal library only > exports. Makes sense for clarity. I guess we should remove the public headers and change the symbol version all at once in '13'? This would instantly make all out of tree providers totally unworkable. Or go back to providing strong compat to the providers as Ben suggested.. In this case someone will have to vote for this with a patch to put the missing symbol back... I know mlx voted for the first path - Ira? Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <20170126180926.GB20330-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <20170126180926.GB20330-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> @ 2017-01-26 18:11 ` Christoph Hellwig [not found] ` <20170126181113.GA8679-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 0 siblings, 1 reply; 21+ messages in thread From: Christoph Hellwig @ 2017-01-26 18:11 UTC (permalink / raw) To: Jason Gunthorpe Cc: Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, ira.weiny On Thu, Jan 26, 2017 at 11:09:26AM -0700, Jason Gunthorpe wrote: > Or go back to providing strong compat to the providers as Ben > suggested.. In this case someone will have to vote for this with a > patch to put the missing symbol back... There should not be any compat - it's just going to drag down development and everyone is free to submit their provider to rdma-core, same as kernel drivers. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <20170126181113.GA8679-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <20170126181113.GA8679-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> @ 2017-01-26 18:27 ` Jason Gunthorpe [not found] ` <20170126182753.GA5056-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-01-26 20:23 ` Leon Romanovsky 1 sibling, 1 reply; 21+ messages in thread From: Jason Gunthorpe @ 2017-01-26 18:27 UTC (permalink / raw) To: Christoph Hellwig Cc: Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, ira.weiny On Thu, Jan 26, 2017 at 10:11:13AM -0800, Christoph Hellwig wrote: > On Thu, Jan 26, 2017 at 11:09:26AM -0700, Jason Gunthorpe wrote: > > Or go back to providing strong compat to the providers as Ben > > suggested.. In this case someone will have to vote for this with a > > patch to put the missing symbol back... > > There should not be any compat - it's just going to drag down > development and everyone is free to submit their provider to rdma-core, > same as kernel drivers. Yep, that was the argument that won the day so far. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <20170126182753.GA5056-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* RE: rdma-core 12 breaks libibverbs ABI [not found] ` <20170126182753.GA5056-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> @ 2017-01-26 20:38 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E132B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 0 siblings, 1 reply; 21+ messages in thread From: Hefty, Sean @ 2017-01-26 20:38 UTC (permalink / raw) To: Jason Gunthorpe, Christoph Hellwig Cc: Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira > > There should not be any compat - it's just going to drag down > > development and everyone is free to submit their provider to rdma- > core, > > same as kernel drivers. > > Yep, that was the argument that won the day so far. I don't disagree with the direction, but I agree with Benjamin. The intent behind a public symbol/structure and their actual usage by applications may differ. Why wasn't the soname updated? - Sean -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1828884A29C6694DAF28B7E6B8A82373AB0E132B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E132B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> @ 2017-01-26 21:49 ` Jason Gunthorpe 0 siblings, 0 replies; 21+ messages in thread From: Jason Gunthorpe @ 2017-01-26 21:49 UTC (permalink / raw) To: Hefty, Sean Cc: Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira On Thu, Jan 26, 2017 at 08:38:40PM +0000, Hefty, Sean wrote: > > > There should not be any compat - it's just going to drag down > > > development and everyone is free to submit their provider to rdma- > > core, > > > same as kernel drivers. > > > > Yep, that was the argument that won the day so far. > > I don't disagree with the direction, but I agree with Benjamin. The > intent behind a public symbol/structure and their actual usage by > applications may differ. Why wasn't the soname updated? Well, no it cannot differ. If apps use private symbols they get burned. Glibc and gcc have established that precedent decades ago. The soname is not bumped because libibverbs is a symver based library, which means we can manage our private ABI safely without ever requiring soname changes. If something does manage to link to it then it gets a runtime dynamic linking failure, not a random crash. Remember, libibverbs fundamentally does not support two copies of the library in the same address space, so one of the major motivations to use a soname to manage version is invalid. Even worse, because of this, if we did bump the soname, it would be very painful and dangerous. So that leaves us with only one point to argue on: Should libibverbs have a private ABI or not? If yes, then we are doing it properly. The issue is muddied a bit because the private ABI has not been as absolutely private as I'd like, so we could add compat for the two symbols and defer the incompat transition till a later release.. This argument would be best supported if someone knows of something that actually breaks out in the wild... Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <20170126181113.GA8679-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2017-01-26 18:27 ` Jason Gunthorpe @ 2017-01-26 20:23 ` Leon Romanovsky [not found] ` <20170126202330.GX6005-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 1 sibling, 1 reply; 21+ messages in thread From: Leon Romanovsky @ 2017-01-26 20:23 UTC (permalink / raw) To: Christoph Hellwig Cc: Jason Gunthorpe, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, ira.weiny [-- Attachment #1: Type: text/plain, Size: 768 bytes --] On Thu, Jan 26, 2017 at 10:11:13AM -0800, Christoph Hellwig wrote: > On Thu, Jan 26, 2017 at 11:09:26AM -0700, Jason Gunthorpe wrote: > > Or go back to providing strong compat to the providers as Ben > > suggested.. In this case someone will have to vote for this with a > > patch to put the missing symbol back... > > There should not be any compat - it's just going to drag down > development and everyone is free to submit their provider to rdma-core, > same as kernel drivers. Agree, I have no desire to support out-of-tree providers. > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <20170126202330.GX6005-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>]
* RE: rdma-core 12 breaks libibverbs ABI [not found] ` <20170126202330.GX6005-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> @ 2017-01-26 23:53 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E15AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 0 siblings, 1 reply; 21+ messages in thread From: Hefty, Sean @ 2017-01-26 23:53 UTC (permalink / raw) To: Leon Romanovsky, Christoph Hellwig Cc: Jason Gunthorpe, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira > > There should not be any compat - it's just going to drag down > > development and everyone is free to submit their provider to rdma- > core, > > same as kernel drivers. > > Agree, I have no desire to support out-of-tree providers. What is the plan for handling live updates to a single vendor library? Is the intent that vendors must now (potentially) provide different versions of their libraries, based on which libibverbs release is deployed by the user? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1828884A29C6694DAF28B7E6B8A82373AB0E15AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E15AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> @ 2017-01-27 1:04 ` Jason Gunthorpe [not found] ` <20170127010452.GA7826-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 0 siblings, 1 reply; 21+ messages in thread From: Jason Gunthorpe @ 2017-01-27 1:04 UTC (permalink / raw) To: Hefty, Sean Cc: Leon Romanovsky, Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira On Thu, Jan 26, 2017 at 11:53:36PM +0000, Hefty, Sean wrote: > > > There should not be any compat - it's just going to drag down > > > development and everyone is free to submit their provider to rdma- > > core, > > > same as kernel drivers. > > > > Agree, I have no desire to support out-of-tree providers. > > What is the plan for handling live updates to a single vendor library? > > Is the intent that vendors must now (potentially) provide different > versions of their libraries, based on which libibverbs release is > deployed by the user? Well, this has been the case for awhile, at least for certain providers (eg mlx5 will not link to older libibverbs). The patch to add IBVERBS_PRIVATE_xx certainly makes it every provider's problem.. As I see it, anyone doing an update driver distribution will have to QA it with a certain libibverbs, so I doubt there is much point in making it too easy for a user to mix and match too widely?? So vendors have a few basic choices: - Ship libibverbs.rpm with the updated driver and matched libibverbs This is pretty easy, I probably would recommend this. - QA the driver with the libibverbs in the various targetted distro releases, as you suggested - Ship their fix to the distro and have the distro push the update It isn't really any different from the problem a vendor would face getting the kernel driver updated.. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <20170127010452.GA7826-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* RE: rdma-core 12 breaks libibverbs ABI [not found] ` <20170127010452.GA7826-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> @ 2017-01-27 1:17 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E1609-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-01-27 21:16 ` Weiny, Ira 1 sibling, 1 reply; 21+ messages in thread From: Hefty, Sean @ 2017-01-27 1:17 UTC (permalink / raw) To: Jason Gunthorpe Cc: Leon Romanovsky, Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira > - Ship libibverbs.rpm with the updated driver and matched libibverbs > This is pretty easy, I probably would recommend this. This has issues if there are multiple providers present. Maybe that doesn't happen in non-testing environments. I wonder if the distros would agree with this approach. > - QA the driver with the libibverbs in the various targetted distro > releases, as you suggested > - Ship their fix to the distro and have the distro push the update > > It isn't really any different from the problem a vendor would face > getting the kernel driver updated.. I agree. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1828884A29C6694DAF28B7E6B8A82373AB0E1609-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E1609-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> @ 2017-01-27 14:32 ` Doug Ledford [not found] ` <41e74ed7-fd8d-89cf-2c79-b99a8d786fac-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 0 siblings, 1 reply; 21+ messages in thread From: Doug Ledford @ 2017-01-27 14:32 UTC (permalink / raw) To: Hefty, Sean, Jason Gunthorpe Cc: Leon Romanovsky, Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira [-- Attachment #1.1: Type: text/plain, Size: 1116 bytes --] On 1/26/2017 8:17 PM, Hefty, Sean wrote: >> - Ship libibverbs.rpm with the updated driver and matched libibverbs >> This is pretty easy, I probably would recommend this. > > This has issues if there are multiple providers present. Maybe that doesn't happen in non-testing environments. I wonder if the distros would agree with this approach. I don't know how much I would rely on the belief that mixed environments don't happen outside of testing. In particular, the structure of Lustre filesystems or the need to gateway from a fabric that one brand of storage server doesn't support to another one would both seem to be times when mixed hardware might exist. For that matter, the typical stock trading RDMA environment mixes IB and Ethernet, maybe in the future it will have OPA in there too? I think you are right that it is the most common, but I don't think we can rely on it being that way all the time. -- Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> GPG Key ID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 884 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <41e74ed7-fd8d-89cf-2c79-b99a8d786fac-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <41e74ed7-fd8d-89cf-2c79-b99a8d786fac-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> @ 2017-01-27 16:22 ` Jason Gunthorpe 2017-01-31 18:37 ` Coulter, Susan K 1 sibling, 0 replies; 21+ messages in thread From: Jason Gunthorpe @ 2017-01-27 16:22 UTC (permalink / raw) To: Doug Ledford Cc: Hefty, Sean, Leon Romanovsky, Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira On Fri, Jan 27, 2017 at 09:32:51AM -0500, Doug Ledford wrote: > On 1/26/2017 8:17 PM, Hefty, Sean wrote: > >> - Ship libibverbs.rpm with the updated driver and matched libibverbs > >> This is pretty easy, I probably would recommend this. > > > > This has issues if there are multiple providers present. Maybe > > that doesn't happen in non-testing environments. I wonder if the > > distros would agree with this approach. > > I don't know how much I would rely on the belief that mixed environments > don't happen outside of testing. Yes, I've heard many complaints in production in this area. As it stands today the changes to libibverbs in vendor trees are too extensive, the only way to resolve them is to merge the vendors patches into one source tree and build a custom release. I think rdma-core will make that much simpler. Making the ABI private or public doesn't really help things either way, as far as I can tell. Remember the stable ABI is upstream, not in a vendor tree, so don't expect a random vendor libibverbs to work with any other providers, even today! They might link, but based on what I've seen they will be subtly broken. Ideally vendors will get their stuff upstream and a site could choose to just use upstream rdma-core instead of a vendor tree. I would still suggest vendors shipping libibverbs.rpm as their update is and reasonable and easy approach for their simple users. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <41e74ed7-fd8d-89cf-2c79-b99a8d786fac-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-01-27 16:22 ` Jason Gunthorpe @ 2017-01-31 18:37 ` Coulter, Susan K [not found] ` <A4803088-726E-4B88-B421-CE1969B6E25B-K3XgeB3KlJxL3NSqSJqx7PIbXMQ5te18@public.gmane.org> 1 sibling, 1 reply; 21+ messages in thread From: Coulter, Susan K @ 2017-01-31 18:37 UTC (permalink / raw) To: Doug Ledford Cc: Hefty, Sean, Jason Gunthorpe, Leon Romanovsky, Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira > On Jan 27, 2017, at 7:32 AM, Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote: > > On 1/26/2017 8:17 PM, Hefty, Sean wrote: >>> - Ship libibverbs.rpm with the updated driver and matched libibverbs >>> This is pretty easy, I probably would recommend this. >> >> This has issues if there are multiple providers present. Maybe that doesn't happen in non-testing environments. I wonder if the distros would agree with this approach. > > I don't know how much I would rely on the belief that mixed environments > don't happen outside of testing. In particular, the structure of Lustre > filesystems or the need to gateway from a fabric that one brand of > storage server doesn't support to another one would both seem to be > times when mixed hardware might exist. For that matter, the typical > stock trading RDMA environment mixes IB and Ethernet, maybe in the > future it will have OPA in there too? This is happening now - not in the future. LANL and other DOE labs already have OPA and IB/mlnx on the same host. > I think you are right that it is > the most common, but I don't think we can rely on it being that way all > the time. > > -- > Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> > GPG Key ID: B826A3330E572FDD > Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD > ================================================== Susan Coulter / HPC-DES Network Capability Lead (505) 667-8425 “Once in a while you get shown the light In the strangest of places if you look at it right” Robert Hunter ================================================== -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <A4803088-726E-4B88-B421-CE1969B6E25B-K3XgeB3KlJxL3NSqSJqx7PIbXMQ5te18@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <A4803088-726E-4B88-B421-CE1969B6E25B-K3XgeB3KlJxL3NSqSJqx7PIbXMQ5te18@public.gmane.org> @ 2017-01-31 20:37 ` Christoph Hellwig 0 siblings, 0 replies; 21+ messages in thread From: Christoph Hellwig @ 2017-01-31 20:37 UTC (permalink / raw) To: Coulter, Susan K Cc: Doug Ledford, Hefty, Sean, Jason Gunthorpe, Leon Romanovsky, Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Weiny, Ira On Tue, Jan 31, 2017 at 06:37:58PM +0000, Coulter, Susan K wrote: > This is happening now - not in the future. > LANL and other DOE labs already have OPA and IB/mlnx on the same host. And have setups with up to three different providers :) But the whole point is that the new rdma-core setup gives us a sane way to handle this: patch your rdma-core tree with whatever vendor updates you need, same as the kernel. The current model where vendors supply a new libibverbs that has all kinds of incompatible hacks is simply not sustainable. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: rdma-core 12 breaks libibverbs ABI [not found] ` <20170127010452.GA7826-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-01-27 1:17 ` Hefty, Sean @ 2017-01-27 21:16 ` Weiny, Ira 1 sibling, 0 replies; 21+ messages in thread From: Weiny, Ira @ 2017-01-27 21:16 UTC (permalink / raw) To: Jason Gunthorpe, Hefty, Sean Cc: Leon Romanovsky, Christoph Hellwig, Benjamin Drung, linux-rdma-u79uwXL29TY76Z2rM5mHXA > > It isn't really any different from the problem a vendor would face getting the > kernel driver updated.. > My concern is that OFA (and the vendors) have not exactly had a good track record of kernel updates in the past. So I'm not sure that this statement give me a lot of comfort. That said, I know that most vendors seem to be making a good effort at integrating upstream. And there is still a mechanism which allows for an external provider. So I'm not opposed to this. Ira -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <20170126173548.GA15006-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2017-01-26 18:09 ` Jason Gunthorpe @ 2017-01-31 12:23 ` Benjamin Drung [not found] ` <1485865383.4257.10.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> 1 sibling, 1 reply; 21+ messages in thread From: Benjamin Drung @ 2017-01-31 12:23 UTC (permalink / raw) To: Christoph Hellwig, Jason Gunthorpe; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig: > On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote: > > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 > > patch > > was merged, but before we release 13 we need to drop all the > > ibv_cmd_* > > entry points from the public headers so that nothing outside rdma- > > core > > can link to these symbols. > > A good thing would be to move all these symbols into a > @IBVERBS_PRIVATE > namespace, similar to what glibc does for it's internal library only > exports. I am fine with moving the private symbols into a IBVERBS_PRIVATE_XX namespace as long as all consumers of the private symbols are build from the rdma-core source package. I need to check if there is a package in Debian that uses the private symbols. So either go forward (extra namespace for private symbols) or revert the ABI breakage (for the time being). The discussion on this list seem to prefer the first option. -- Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Geschäftsführer: Andreas Gauger, Achim Weiss. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1485865383.4257.10.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <1485865383.4257.10.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> @ 2017-01-31 15:06 ` Benjamin Drung [not found] ` <1485875171.4257.18.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> 2017-01-31 16:38 ` Jason Gunthorpe 1 sibling, 1 reply; 21+ messages in thread From: Benjamin Drung @ 2017-01-31 15:06 UTC (permalink / raw) To: linux-rdma-u79uwXL29TY76Z2rM5mHXA Am Dienstag, den 31.01.2017, 13:23 +0100 schrieb Benjamin Drung: > Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig: > > On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote: > > > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 > > > patch > > > was merged, but before we release 13 we need to drop all the > > > ibv_cmd_* > > > entry points from the public headers so that nothing outside > > > rdma- > > > core > > > can link to these symbols. > > > > A good thing would be to move all these symbols into a > > @IBVERBS_PRIVATE > > namespace, similar to what glibc does for it's internal library > > only > > exports. > > I am fine with moving the private symbols into a IBVERBS_PRIVATE_XX > namespace as long as all consumers of the private symbols are build > from the rdma-core source package. I need to check if there is a > package in Debian that uses the private symbols. I brought this question to the debian-devel mailing list: https://lists.debian.org/debian-devel/2017/01/msg00791.html One private feedback was that removing symbols without bumping the soname is a really bad idea (even if they are private symbols). So one option would be to bump the soname once and remove all private symbols in that run. I searched for "ibv_cmd_" in Debian's source code: https://codesearch.debian.net/search?q=ibv_cmd_ It boils down to these source packages: shipped by rdma-core ==================== * librdmacm * libibverbs provided by ibverbs-providers ============================= * libcxgb3 * libmlx5 * libmthca * libnes * libipathverbs * libmlx4 remaining ========= * openmpi openmpi uses ibv_cmd_open_xrcd in config/opal_check_openfabrics.m4: # We have to check functions both exits *and* are declared # since some distro ship broken ibverbs devel headers # IBV_DEVICE_XRC is common to all OFED versions # ibv_create_xrc_rcv_qp was added in OFED 1.3 # ibv_cmd_open_xrcd (aka XRC Domains) was added in OFED 3.12 So I recommend to contact openmpi to let them change their configure check to not check for ibv_cmd_open_xrcd any more. -- Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Geschäftsführer: Andreas Gauger, Achim Weiss. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1485875171.4257.18.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>]
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <1485875171.4257.18.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> @ 2017-01-31 17:35 ` Jason Gunthorpe 0 siblings, 0 replies; 21+ messages in thread From: Jason Gunthorpe @ 2017-01-31 17:35 UTC (permalink / raw) To: Benjamin Drung; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA On Tue, Jan 31, 2017 at 04:06:11PM +0100, Benjamin Drung wrote: > One private feedback was that removing symbols without bumping the > soname is a really bad idea (even if they are private symbols). So one > option would be to bump the soname once and remove all private symbols > in that run. Bumping the soname is also a really bad idea - we need to pick which one is less bad :) > I searched for "ibv_cmd_" in Debian's source code: > https://codesearch.debian.net/search?q=ibv_cmd_ > It boils down to these source packages: Neat. I also looked at the removed headerfiles which will cause FTBS. dmtcp seems to include an old wrong copy of libibverbs/examples/devinfo.c - I have no idea why. usr/include/infiniband/arch.h dpdk dmtcp fio qlvnictools These seem to be using htonll/etc (replace with glibc's htobe64) usr/include/infiniband/driver.h openmpi dmtcp libfabric Most of these are probably unnecessary.. infiniband/marshall.h has none > openmpi uses ibv_cmd_open_xrcd in config/opal_check_openfabrics.m4: > > # We have to check functions both exits *and* are declared > # since some distro ship broken ibverbs devel headers > # IBV_DEVICE_XRC is common to all OFED versions > # ibv_create_xrc_rcv_qp was added in OFED 1.3 > # ibv_cmd_open_xrcd (aka XRC Domains) was added in OFED 3.12 > > So I recommend to contact openmpi to let them change their configure > check to not check for ibv_cmd_open_xrcd any more. Yes, they will need to make a few small changes This is a very small amount of usage and other than the htonll stuff, none of it looks at all right to me. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: rdma-core 12 breaks libibverbs ABI [not found] ` <1485865383.4257.10.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> 2017-01-31 15:06 ` Benjamin Drung @ 2017-01-31 16:38 ` Jason Gunthorpe 1 sibling, 0 replies; 21+ messages in thread From: Jason Gunthorpe @ 2017-01-31 16:38 UTC (permalink / raw) To: Benjamin Drung, Doug Ledford Cc: Christoph Hellwig, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Tue, Jan 31, 2017 at 01:23:03PM +0100, Benjamin Drung wrote: > Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig: > > On Thu, Jan 26, 2017 at 10:13:25AM -0700, Jason Gunthorpe wrote: > > > I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 > > > patch > > > was merged, but before we release 13 we need to drop all the > > > ibv_cmd_* > > > entry points from the public headers so that nothing outside rdma- > > > core > > > can link to these symbols. > > > > A good thing would be to move all these symbols into a > > @IBVERBS_PRIVATE > > namespace, similar to what glibc does for it's internal library only > > exports. > > I am fine with moving the private symbols into a IBVERBS_PRIVATE_XX > namespace as long as all consumers of the private symbols are build > from the rdma-core source package. I need to check if there is a > package in Debian that uses the private symbols. > So either go forward (extra namespace for private symbols) or revert > the ABI breakage (for the time being). The discussion on this list seem > to prefer the first option. Yes. We must implement one of the two options before releasing '13' I have sent patches on github to do the first.. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2017-01-31 20:37 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-01-26 15:55 rdma-core 12 breaks libibverbs ABI Benjamin Drung [not found] ` <1485446147.4194.4.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> 2017-01-26 17:13 ` Jason Gunthorpe [not found] ` <20170126171325.GA16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-01-26 17:35 ` Christoph Hellwig [not found] ` <20170126173548.GA15006-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2017-01-26 18:09 ` Jason Gunthorpe [not found] ` <20170126180926.GB20330-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-01-26 18:11 ` Christoph Hellwig [not found] ` <20170126181113.GA8679-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2017-01-26 18:27 ` Jason Gunthorpe [not found] ` <20170126182753.GA5056-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-01-26 20:38 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E132B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-01-26 21:49 ` Jason Gunthorpe 2017-01-26 20:23 ` Leon Romanovsky [not found] ` <20170126202330.GX6005-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-01-26 23:53 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E15AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-01-27 1:04 ` Jason Gunthorpe [not found] ` <20170127010452.GA7826-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-01-27 1:17 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A82373AB0E1609-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-01-27 14:32 ` Doug Ledford [not found] ` <41e74ed7-fd8d-89cf-2c79-b99a8d786fac-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-01-27 16:22 ` Jason Gunthorpe 2017-01-31 18:37 ` Coulter, Susan K [not found] ` <A4803088-726E-4B88-B421-CE1969B6E25B-K3XgeB3KlJxL3NSqSJqx7PIbXMQ5te18@public.gmane.org> 2017-01-31 20:37 ` Christoph Hellwig 2017-01-27 21:16 ` Weiny, Ira 2017-01-31 12:23 ` Benjamin Drung [not found] ` <1485865383.4257.10.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> 2017-01-31 15:06 ` Benjamin Drung [not found] ` <1485875171.4257.18.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> 2017-01-31 17:35 ` Jason Gunthorpe 2017-01-31 16:38 ` Jason Gunthorpe
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.