rdma-core build environment enabling out-of-core providers

rdma-core build environment enabling out-of-core providers

[Steve Wise]

Currently there is no way to build an out-of-rdma-core provider on a system
installed with rdma-core.  It would be useful to have rdma-core install the
needed headers and libs (ie the ccan stuff) as part of its install.  This would
be similar to previous releases of libibverbs that included the headers needed
for providers to build.   I would like to implement this for rdma-core, but
wanted feedback on this approach.  Maybe an rdma-core-devel type package?  I
guess I'm thinking this would be analogous to building out-of-kernel modules,
which is available for kernel developers. 

Thoughts?

Thanks,

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Jason Gunthorpe]

On Wed, Aug 02, 2017 at 08:55:06AM -0500, Steve Wise wrote:

> Currently there is no way to build an out-of-rdma-core provider on a
> system installed with rdma-core.  It would be useful to have
> rdma-core install the needed headers and libs (ie the ccan stuff) as
> part of its install.  This would be similar to previous releases of
> libibverbs that included the headers needed for providers to build.
> I would like to implement this for rdma-core, but wanted feedback on
> this approach.  Maybe an rdma-core-devel type package?  I guess I'm
> thinking this would be analogous to building out-of-kernel modules,
> which is available for kernel developers.

I don't think that really works. eg the util/mmio.h relies on
stdatomic.h to work, which means it relies on the cmake shim to
provide stdatomic.h on old systems. Other private headers may be similar
in their integration with cmake/etc.

You can't really publish those headers sanely, and part of the point
here was specifically to allow using things like statomic.h properly
without having to be concerned about how to install the shim into a
public header.

In my view, the best approach, which I have suggested before, is to
build the out-of-tree stuff 'in-tree' and discard everything except
the single provider library you are building for. For instance, a rpm
spec file that produces as contents a single provider shlib.

This seems like the least amount of work for everyone, and it doesn't
really matter, I think, that the 'source' for the 'provider' carries
lots of unbuilt stuff.

Perhaps the best way to accomplish that is to add a sample rpm spec
file to Documentation/ or something that shows how to do that. If you
want to take a first stab at it I can give you some help.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core build environment enabling out-of-core providers

[Steve Wise]

> > Currently there is no way to build an out-of-rdma-core provider on a
> > system installed with rdma-core.  It would be useful to have
> > rdma-core install the needed headers and libs (ie the ccan stuff) as
> > part of its install.  This would be similar to previous releases of
> > libibverbs that included the headers needed for providers to build.
> > I would like to implement this for rdma-core, but wanted feedback on
> > this approach.  Maybe an rdma-core-devel type package?  I guess I'm
> > thinking this would be analogous to building out-of-kernel modules,
> > which is available for kernel developers.
> 
> I don't think that really works. eg the util/mmio.h relies on
> stdatomic.h to work, which means it relies on the cmake shim to
> provide stdatomic.h on old systems. Other private headers may be similar
> in their integration with cmake/etc.
> 
> You can't really publish those headers sanely, and part of the point
> here was specifically to allow using things like statomic.h properly
> without having to be concerned about how to install the shim into a
> public header.
> 
> In my view, the best approach, which I have suggested before, is to
> build the out-of-tree stuff 'in-tree' and discard everything except
> the single provider library you are building for. For instance, a rpm
> spec file that produces as contents a single provider shlib.
> 
> This seems like the least amount of work for everyone, and it doesn't
> really matter, I think, that the 'source' for the 'provider' carries
> lots of unbuilt stuff.
> 
> Perhaps the best way to accomplish that is to add a sample rpm spec
> file to Documentation/ or something that shows how to do that. If you
> want to take a first stab at it I can give you some help.

Hey Jason, Thanks for this detailed response!  I appreciate it.  

Today I did two experiments:

1) I took my out-of-tree provider, and moved it into rdma-core, updated it to
use the rdma-core bldenv and APIs.  This included the rdma-core mmio and
dma/cache services, plus a few other nits like ccan, and the new byte swapping
API, and got it building.  That was pretty straight forward.

2) I took the updated out-of-tree provider src from 1), and put it back in the
out-of-tree provider automake/config bldenv, and then tried to pull in whatever
was needed from rdma-core to build it.  This resulted in me pulling certain
files, like you mentioned above.  I just pulled them from the rdma-core/build/
tree for the sake of getting it to build.   Basically I needed ccan/*
infiniband/driver.h, and util/compiler.h/udma_barrier.h.  That got it to build.


I haven't tested either of these yet to see if they actually load.  But that's
where I'm at.  #2 has the issues you describe, in that the atomic header is
auto-generated based on the installed distro/platform.  So that leads to using
#1 which is what you recommend.  I'm ok with using #1, but then the issue
becomes supporting my provider across older non rdma-core installed systems.
Ideally, I want to be able to build my provider to install into a RHEL7.x
system, for example, which uses the older libibverbs.  But I also want it to
build/install into a system installed with OFED-4.8 which includes rdma-core-14
I think, as well as SLE12SP3 which has rdma-core as part of its distribution as
well.

I guess the issue is how to use 1) but allow a build/install into an older
libibverbs system.   I don't think it is possible w/o a hybrid bldenv that uses
rdma-core bldenv if that is what is installed, or the old libibverbs
headers/APIs if that is installed. 

And I'm trying to do this cleanly :)  (backporting is rarely clean)

Thanks,

Steve




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Jason Gunthorpe]

On Wed, Aug 02, 2017 at 02:16:09PM -0500, Steve Wise wrote:
> 1) I took my out-of-tree provider, and moved it into rdma-core, updated it to
> use the rdma-core bldenv and APIs.  This included the rdma-core mmio and
> dma/cache services, plus a few other nits like ccan, and the new byte swapping
> API, and got it building.  That was pretty straight forward.

Great, that is what I expected..

> 2) I took the updated out-of-tree provider src from 1), and put it back in the
> out-of-tree provider automake/config bldenv, and then tried to pull in whatever
> was needed from rdma-core to build it.  This resulted in me pulling certain
> files, like you mentioned above.  I just pulled them from the rdma-core/build/
> tree for the sake of getting it to build.   Basically I needed ccan/*
> infiniband/driver.h, and util/compiler.h/udma_barrier.h.  That got it to build.

Just be careful that all the cmake magic is working properly in your
other environment, for all the distros you want to target..

> I haven't tested either of these yet to see if they actually load.  But that's
> where I'm at.  #2 has the issues you describe, in that the atomic header is
> auto-generated based on the installed distro/platform.  So that leads to using
> #1 which is what you recommend.  I'm ok with using #1, but then the issue
> becomes supporting my provider across older non rdma-core installed
> systems.

You can handle this by rolling back the rdma-core repository to the
right points in history and re-instering your provider. This will
require some level of cherry picking future patches (eg ccan, etc),
but may be the simplest approach for this specific requirement.

This would be somewhere around
0c0914e1e9b7a68bcebfe785b6df30500ca7d2e0 for RHEL7 libibverbs.

You can find a similar point for OFED.

This could be automated, so you can develope against upstream, and
then using a script create a rdma-core with the classic private ABI,
and copy the provider from the upstream to build it.

Alternatively, just have your customers use a new libibverbs on
RHEL. That reduces your QA burden and development workload...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core build environment enabling out-of-core providers

[Steve Wise]

> > 2) I took the updated out-of-tree provider src from 1), and put it back in
the
> > out-of-tree provider automake/config bldenv, and then tried to pull in
whatever
> > was needed from rdma-core to build it.  This resulted in me pulling certain
> > files, like you mentioned above.  I just pulled them from the
rdma-core/build/
> > tree for the sake of getting it to build.   Basically I needed ccan/*
> > infiniband/driver.h, and util/compiler.h/udma_barrier.h.  That got it to
build.
> 
> Just be careful that all the cmake magic is working properly in your
> other environment, for all the distros you want to target..

Well definitely no magic is happening in my specific experiment, since I just
copied the headers from the built rdma-core tree which means they were
customized for RHEL7.2, which is what was installed.  But I just wanted to get a
feel for what files exactly were needed to build a provider.  I suppose I could
take that cmake magic into my bldenv and just the src files needed to produce
the auto-generated files.  Is it just the stdatomic.h file that is tweaked?

> 
> > I haven't tested either of these yet to see if they actually load.  But
that's
> > where I'm at.  #2 has the issues you describe, in that the atomic header is
> > auto-generated based on the installed distro/platform.  So that leads to
using
> > #1 which is what you recommend.  I'm ok with using #1, but then the issue
> > becomes supporting my provider across older non rdma-core installed
> > systems.
> 
> You can handle this by rolling back the rdma-core repository to the
> right points in history and re-instering your provider. This will
> require some level of cherry picking future patches (eg ccan, etc),
> but may be the simplest approach for this specific requirement.
> 
> This would be somewhere around
> 0c0914e1e9b7a68bcebfe785b6df30500ca7d2e0 for RHEL7 libibverbs.
> 
> You can find a similar point for OFED.
> 

Hmm.  This is interesting.

> This could be automated, so you can develope against upstream, and
> then using a script create a rdma-core with the classic private ABI,
> and copy the provider from the upstream to build it.
> 
> Alternatively, just have your customers use a new libibverbs on
> RHEL. That reduces your QA burden and development workload...

If my package installs libibverbs along with my provider, then potentially I
break any existing providers installed with whatever rdma-core, ofed, or inboxed
libibverbs that was installed.   I'm not sure if this is acceptable, but I agree
it simplifies the QA matrix.  

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Jason Gunthorpe]

On Wed, Aug 02, 2017 at 03:05:31PM -0500, Steve Wise wrote:

> > Just be careful that all the cmake magic is working properly in your
> > other environment, for all the distros you want to target..
> 
> Well definitely no magic is happening in my specific experiment, since I just
> copied the headers from the built rdma-core tree which means they were
> customized for RHEL7.2, which is what was installed.  But I just wanted to get a
> feel for what files exactly were needed to build a provider.  I suppose I could
> take that cmake magic into my bldenv and just the src files needed to produce
> the auto-generated files.  Is it just the stdatomic.h file that is tweaked?

valgrind, stdatomic and some of the kernel are currently shimed, plus
some definitions in config.h that control a few things in compiler.h
and some of the ccan headers. The trick is this can change, and
future failure might be silent..

> > This could be automated, so you can develope against upstream, and
> > then using a script create a rdma-core with the classic private ABI,
> > and copy the provider from the upstream to build it.
> > 
> > Alternatively, just have your customers use a new libibverbs on
> > RHEL. That reduces your QA burden and development workload...
> 
> If my package installs libibverbs along with my provider, then
> potentially I break any existing providers installed with whatever
> rdma-core, ofed, or inboxed libibverbs that was installed.  I'm not
> sure if this is acceptable, but I agree it simplifies the QA matrix.

In this case install upstream rdma-core providers along side yours..

Also, I just looked for you, and found this:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/7.4_Release_Notes/new_features_networking.html#BZ1404035

So RHEL 7.4 is shipping rdma-core 13 apparently ?

So replacing RHEL 7.4's rdma-core 13 with 15+your stuff is pretty
straightforward, IHMO.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

On Wed, Aug 02, 2017 at 02:23:00PM -0600, Jason Gunthorpe wrote:
> On Wed, Aug 02, 2017 at 03:05:31PM -0500, Steve Wise wrote:
>
> Also, I just looked for you, and found this:
>
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/7.4_Release_Notes/new_features_networking.html#BZ1404035
>
> So RHEL 7.4 is shipping rdma-core 13 apparently ?

Yes, the Debian/Ubuntu are the last distro which still ship libibverbs.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rdma-core build environment enabling out-of-core providers

[Nicolas Morey-Chaisemartin]

Le 02/08/2017 à 21:16, Steve Wise a écrit :
>>> Currently there is no way to build an out-of-rdma-core provider on a
>>> system installed with rdma-core.  It would be useful to have
>>> rdma-core install the needed headers and libs (ie the ccan stuff) as
>>> part of its install.  This would be similar to previous releases of
>>> libibverbs that included the headers needed for providers to build.
>>> I would like to implement this for rdma-core, but wanted feedback on
>>> this approach.  Maybe an rdma-core-devel type package?  I guess I'm
>>> thinking this would be analogous to building out-of-kernel modules,
>>> which is available for kernel developers.
>> I don't think that really works. eg the util/mmio.h relies on
>> stdatomic.h to work, which means it relies on the cmake shim to
>> provide stdatomic.h on old systems. Other private headers may be similar
>> in their integration with cmake/etc.
>>
>> You can't really publish those headers sanely, and part of the point
>> here was specifically to allow using things like statomic.h properly
>> without having to be concerned about how to install the shim into a
>> public header.
>>
>> In my view, the best approach, which I have suggested before, is to
>> build the out-of-tree stuff 'in-tree' and discard everything except
>> the single provider library you are building for. For instance, a rpm
>> spec file that produces as contents a single provider shlib.
>>
>> This seems like the least amount of work for everyone, and it doesn't
>> really matter, I think, that the 'source' for the 'provider' carries
>> lots of unbuilt stuff.
>>
>> Perhaps the best way to accomplish that is to add a sample rpm spec
>> file to Documentation/ or something that shows how to do that. If you
>> want to take a first stab at it I can give you some help.
> Hey Jason, Thanks for this detailed response!  I appreciate it.  
>
> Today I did two experiments:
>
> 1) I took my out-of-tree provider, and moved it into rdma-core, updated it to
> use the rdma-core bldenv and APIs.  This included the rdma-core mmio and
> dma/cache services, plus a few other nits like ccan, and the new byte swapping
> API, and got it building.  That was pretty straight forward.
>
> 2) I took the updated out-of-tree provider src from 1), and put it back in the
> out-of-tree provider automake/config bldenv, and then tried to pull in whatever
> was needed from rdma-core to build it.  This resulted in me pulling certain
> files, like you mentioned above.  I just pulled them from the rdma-core/build/
> tree for the sake of getting it to build.   Basically I needed ccan/*
> infiniband/driver.h, and util/compiler.h/udma_barrier.h.  That got it to build.
>
>
> I haven't tested either of these yet to see if they actually load.  But that's
> where I'm at.  #2 has the issues you describe, in that the atomic header is
> auto-generated based on the installed distro/platform.  So that leads to using
> #1 which is what you recommend. 
Hi,

I 'm not sure I see the issue here (or at least nothing easily fixed).
If the standard spec produces an extra rdma-core-provider-devel with those headers, it's just a matter of rebuilding the out-of-tree provider against the right distro/package.

Nicolas

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core build environment enabling out-of-core providers

[Steve Wise]

> >>
> >> You can't really publish those headers sanely, and part of the point
> >> here was specifically to allow using things like statomic.h properly
> >> without having to be concerned about how to install the shim into a
> >> public header.
> >>
> >> In my view, the best approach, which I have suggested before, is to
> >> build the out-of-tree stuff 'in-tree' and discard everything except
> >> the single provider library you are building for. For instance, a rpm
> >> spec file that produces as contents a single provider shlib.
> >>
> >> This seems like the least amount of work for everyone, and it doesn't
> >> really matter, I think, that the 'source' for the 'provider' carries
> >> lots of unbuilt stuff.
> >>
> >> Perhaps the best way to accomplish that is to add a sample rpm spec
> >> file to Documentation/ or something that shows how to do that. If you
> >> want to take a first stab at it I can give you some help.
> > Hey Jason, Thanks for this detailed response!  I appreciate it.
> >
> > Today I did two experiments:
> >
> > 1) I took my out-of-tree provider, and moved it into rdma-core, updated it to
> > use the rdma-core bldenv and APIs.  This included the rdma-core mmio and
> > dma/cache services, plus a few other nits like ccan, and the new byte swapping
> > API, and got it building.  That was pretty straight forward.
> >
> > 2) I took the updated out-of-tree provider src from 1), and put it back in the
> > out-of-tree provider automake/config bldenv, and then tried to pull in whatever
> > was needed from rdma-core to build it.  This resulted in me pulling certain
> > files, like you mentioned above.  I just pulled them from the rdma-core/build/
> > tree for the sake of getting it to build.   Basically I needed ccan/*
> > infiniband/driver.h, and util/compiler.h/udma_barrier.h.  That got it to build.
> >
> >
> > I haven't tested either of these yet to see if they actually load.  But that's
> > where I'm at.  #2 has the issues you describe, in that the atomic header is
> > auto-generated based on the installed distro/platform.  So that leads to using
> > #1 which is what you recommend.
> Hi,
> 
> I 'm not sure I see the issue here (or at least nothing easily fixed).
> If the standard spec produces an extra rdma-core-provider-devel with those
> headers, it's just a matter of rebuilding the out-of-tree provider against the right
> distro/package.
> 

After thinking about this more, I agree:  If rdma-core is built into binary platform and distro-specific rpms, then these header files are definitely available to export as a -devel rpm for that same platform/distro.  

Jason, am I missing something?

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Jason Gunthorpe]

On Thu, Aug 03, 2017 at 08:48:50AM -0500, Steve Wise wrote:

> > I 'm not sure I see the issue here (or at least nothing easily
> > fixed).  If the standard spec produces an extra
> > rdma-core-provider-devel with those headers, it's just a matter of
> > rebuilding the out-of-tree provider against the right
> > distro/package.
> 
> After thinking about this more, I agree: If rdma-core is built into
> binary platform and distro-specific rpms, then these header files
> are definitely available to export as a -devel rpm for that same
> platform/distro.

The issue is where do you put headers like stdatomic.h, config.h, etc.

These headers will collide with normal headers in various cases..

It is designed to be built with a private include/ that relies on
#include_next to make everything work out, you cannot just dump the
headers in /usr/include.

If instead you do something like /usr/share/rdma-core/include then you
still have the problem of conflicting with common user header names
like config.h, and the problem of being matched exactly to the
compiler that cmake was using when it made the headers.

Overall it is a very fragile approach..

There are also minor issues like actually building and linking the
provider correctly, there were many issues there pre-rdma-core,
building it outside the existing cmake framework means someone else
has to actually understand how to do this properly :)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Nicolas Morey-Chaisemartin]

Le 03/08/2017 à 16:47, Jason Gunthorpe a écrit :
> On Thu, Aug 03, 2017 at 08:48:50AM -0500, Steve Wise wrote:
>
>>> I 'm not sure I see the issue here (or at least nothing easily
>>> fixed).  If the standard spec produces an extra
>>> rdma-core-provider-devel with those headers, it's just a matter of
>>> rebuilding the out-of-tree provider against the right
>>> distro/package.
>> After thinking about this more, I agree: If rdma-core is built into
>> binary platform and distro-specific rpms, then these header files
>> are definitely available to export as a -devel rpm for that same
>> platform/distro.
> The issue is where do you put headers like stdatomic.h, config.h, etc.
>
> These headers will collide with normal headers in various cases..
>
> It is designed to be built with a private include/ that relies on
> #include_next to make everything work out, you cannot just dump the
> headers in /usr/include.
>
> If instead you do something like /usr/share/rdma-core/include then you
> still have the problem of conflicting with common user header names
> like config.h, and the problem of being matched exactly to the
> compiler that cmake was using when it made the headers.
>
> Overall it is a very fragile approach..
>
> There are also minor issues like actually building and linking the
> provider correctly, there were many issues there pre-rdma-core,
> building it outside the existing cmake framework means someone else
> has to actually understand how to do this properly :)
>
> Jason
I haven't looked into the build system but isn't this something the devel rpm could provide ?
This means not only packaging the header (a non standard include path would be the right solution I think but also the necessary cmake plumbing so it becomes plug and play for a provider to build its sources. Something along the lines of the way kernel allows you to build your out of tree modules.

If this is done upstream once and for all, and not by every out-of-tree provider, it is not fragile anymore.

Nicolas

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Jason Gunthorpe]

On Thu, Aug 03, 2017 at 05:01:16PM +0200, Nicolas Morey-Chaisemartin wrote:

> I haven't looked into the build system but isn't this something the
> devel rpm could provide ?

Conceptually, perhaps, but someone has to figure out how to split out
the necessary cmake bits into something reusable, it is no longer
entirely trivial :)

Also don't forget that the internal API inside the header files
changes, a provider that has been modified for rdma core 15 will not
compile on rdma core 13, just like the kernel..

I just wonder if it is worth the all the proposed work, I expect out
of tree providers to be very rare and temporary things.

> If this is done upstream once and for all, and not by every
> out-of-tree provider, it is not fragile anymore.

Someone would also have to contribute some kind of test infrastructure
to keep it working for upstream.. If it doesn't run in cbuild I don't
test it :P

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Nicolas Morey-Chaisemartin]

Le 03/08/2017 à 19:07, Jason Gunthorpe a écrit :
> On Thu, Aug 03, 2017 at 05:01:16PM +0200, Nicolas Morey-Chaisemartin wrote:
>
>> I haven't looked into the build system but isn't this something the
>> devel rpm could provide ?
> Conceptually, perhaps, but someone has to figure out how to split out
> the necessary cmake bits into something reusable, it is no longer
> entirely trivial :)
Yes, this is going to take some time.

> Also don't forget that the internal API inside the header files
> changes, a provider that has been modified for rdma core 15 will not
> compile on rdma core 13, just like the kernel..
I guess there are enough VERSION defines in rdma-core to make the provider portable.
But yes, supporting multiple versions is a lot more work.

> I just wonder if it is worth the all the proposed work, I expect out
> of tree providers to be very rare and temporary things.

The lazy (smart?) thing to do here is to wait for an out-of-tree provider to need it (Steve ?) and give them some support to do it.
If they can't be bothered, it's probably not worth it. If they want it, they'll do most of the work and the feature gets upstreamed.

>
>> If this is done upstream once and for all, and not by every
>> out-of-tree provider, it is not fragile anymore.
> Someone would also have to contribute some kind of test infrastructure
> to keep it working for upstream.. If it doesn't run in cbuild I don't
> test it :P

It shouldn't be too hard to automatically extract one of the in-tree provider and try to compile it as an out-of-tree.
And this way it's always up-to-date for testing.

Nicolas

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core build environment enabling out-of-core providers

[Steve Wise]

> 
> Le 03/08/2017 à 19:07, Jason Gunthorpe a écrit :
> > On Thu, Aug 03, 2017 at 05:01:16PM +0200, Nicolas Morey-Chaisemartin
> wrote:
> >
> >> I haven't looked into the build system but isn't this something the
> >> devel rpm could provide ?
> > Conceptually, perhaps, but someone has to figure out how to split out
> > the necessary cmake bits into something reusable, it is no longer
> > entirely trivial :)
> Yes, this is going to take some time.
> 
> > Also don't forget that the internal API inside the header files
> > changes, a provider that has been modified for rdma core 15 will not
> > compile on rdma core 13, just like the kernel..
> I guess there are enough VERSION defines in rdma-core to make the
> provider portable.
> But yes, supporting multiple versions is a lot more work.
> 

I'm not sure the rdma-core needs to worry about this, other than to provide a compile-time way to detect the version.  IE I don't think rdma-core v15 should bother supporting out-of-tree providers that don't support the v15 provider<->libibverbs API.  It would be up to the out-of-tree providers to support both v13 and v15, for example. 

> > I just wonder if it is worth the all the proposed work, I expect out
> > of tree providers to be very rare and temporary things.
> 
> The lazy (smart?) thing to do here is to wait for an out-of-tree provider to
> need it (Steve ?) and give them some support to do it.
> If they can't be bothered, it's probably not worth it. If they want it, they'll
> do most of the work and the feature gets upstreamed.
> 

Oh I'm already bothered.  😊  I will sign up for this.  If I fall in and sink, hopefully Jason will fish me out of the drink.  Cmake is new to me.

> >
> >> If this is done upstream once and for all, and not by every
> >> out-of-tree provider, it is not fragile anymore.
> > Someone would also have to contribute some kind of test infrastructure
> > to keep it working for upstream.. If it doesn't run in cbuild I don't
> > test it :P
> 
> It shouldn't be too hard to automatically extract one of the in-tree provider
> and try to compile it as an out-of-tree.
> And this way it's always up-to-date for testing.
> 

That's a good idea.  We could also have a simple "noop" type provider that at least registers with the core but never claims any devices.  This would at least verify the build/install, and provider-load by libibverbs.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core build environment enabling out-of-core providers

[Nicolas Morey-Chaisemartin]

Le 04/08/2017 à 13:30, Steve Wise a écrit :
>> Le 03/08/2017 à 19:07, Jason Gunthorpe a écrit :
>>> On Thu, Aug 03, 2017 at 05:01:16PM +0200, Nicolas Morey-Chaisemartin
>> wrote:
>>>> I haven't looked into the build system but isn't this something the
>>>> devel rpm could provide ?
>>> Conceptually, perhaps, but someone has to figure out how to split out
>>> the necessary cmake bits into something reusable, it is no longer
>>> entirely trivial :)
>> Yes, this is going to take some time.
>>
>>> Also don't forget that the internal API inside the header files
>>> changes, a provider that has been modified for rdma core 15 will not
>>> compile on rdma core 13, just like the kernel..
>> I guess there are enough VERSION defines in rdma-core to make the
>> provider portable.
>> But yes, supporting multiple versions is a lot more work.
>>
> I'm not sure the rdma-core needs to worry about this, other than to provide a compile-time way to detect the version.  IE I don't think rdma-core v15 should bother supporting out-of-tree providers that don't support the v15 provider<->libibverbs API.  It would be up to the out-of-tree providers to support both v13 and v15, for example. 
It's just a matter of exporting the PACKAGE_VERSION from the CMakeLists to one the future header to install.
It's not required, but I guess it'll make the job of out-of-tree providers easier if they can just adapt to the proper API depending on this.

>>> I just wonder if it is worth the all the proposed work, I expect out
>>> of tree providers to be very rare and temporary things.
>> The lazy (smart?) thing to do here is to wait for an out-of-tree provider to
>> need it (Steve ?) and give them some support to do it.
>> If they can't be bothered, it's probably not worth it. If they want it, they'll
>> do most of the work and the feature gets upstreamed.
>>
> Oh I'm already bothered.  😊  I will sign up for this.  If I fall in and sink, hopefully Jason will fish me out of the drink.  Cmake is new to me.
>
>>>> If this is done upstream once and for all, and not by every
>>>> out-of-tree provider, it is not fragile anymore.
>>> Someone would also have to contribute some kind of test infrastructure
>>> to keep it working for upstream.. If it doesn't run in cbuild I don't
>>> test it :P
>> It shouldn't be too hard to automatically extract one of the in-tree provider
>> and try to compile it as an out-of-tree.
>> And this way it's always up-to-date for testing.
>>
> That's a good idea.  We could also have a simple "noop" type provider that at least registers with the core but never claims any devices.  This would at least verify the build/install, and provider-load by libibverbs.
>
> Steve.
>

noop is good for testing. A true provider would make sure we haven't missed any headers.

Nicolas

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core build environment enabling out-of-core providers

[Steve Wise]

> >> It shouldn't be too hard to automatically extract one of the in-tree provider
> >> and try to compile it as an out-of-tree.
> >> And this way it's always up-to-date for testing.
> >>
> > That's a good idea.  We could also have a simple "noop" type provider that at
> least registers with the core but never claims any devices.  This would at least verify
> the build/install, and provider-load by libibverbs.
> >
> > Steve.
> >
> 
> noop is good for testing. A true provider would make sure we haven't missed any
> headers.

Agreed.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html