Re: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV

Re: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1939 bytes --]

Hi Lance,

On Fri, 2018-07-20 at 16:34 -0500, Lance Hartmann ORACLE wrote:
> Another especially notable take-away for me was the realization that the
> environment library -- I'll refer to the default env for example here,
> libspdk_env_dpdk.a -- consists only of the objects compiled from
> SPDK_ROOT_DIR/lib/env_dpdk.   With the ability to specify an alternate
> environment, semantically, I had mistakenly assumed that the env library would
> contain not only the SPDK implementation of the environment API, but also
> everything on which it depended; i.e. for the default environment, the DPDK
> objects.   But, that doesn't appear to be so.   Instead, when performing final
> linking of each SPDK executable, not only is the SPDK environment library
> specified, but so are the necessary DPDK libs on which it depends.   Variables
> pulled in from the SPDK's environment makefile, env.mk, are used to specify
> those DPDK libs along with additional, special linking flags.
> 
> We need to update the doc's Porting Guide to reflect these details, and I'd be
> happy to volunteer to work on that effort. 

Please do!

> However, before embarking on that task, I'd like to pose the question:   are
> there reasons why we couldn't or shouldn't produce the environment library to
> consist of both the SPDK env implementation layer and the objects on which it
> depends?  It would seem that would make final linking of executables a little
> easier and reduce the complexity/effort of someone wanting to develop and use
> an alternate environment.

How do you envision this playing with environment libraries that need to link
against shared libraries, especially ones provided with the system itself? I'm
not an expert in all of the available linker options, so maybe there is some way
to embed enough information to properly link/load a shared library directly into
the static library.

Thanks,
Ben

Re: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV

[Lance Hartmann ORACLE]

[-- Attachment #1: Type: text/plain, Size: 5627 bytes --]


> On Aug 1, 2018, at 3:51 PM, Walker, Benjamin <benjamin.walker(a)intel.com> wrote:
> 
> On Mon, 2018-07-23 at 13:46 -0500, Lance Hartmann ORACLE wrote:
>>> On Jul 20, 2018, at 5:17 PM, Walker, Benjamin <benjamin.walker(a)intel.com> wrote:
>>> 
>>> On Fri, 2018-07-20 at 16:34 -0500, Lance Hartmann ORACLE wrote:
>>>> Another especially notable take-away for me was the realization that the
>>>> environment library -- I'll refer to the default env for example here,
>>>> libspdk_env_dpdk.a -- consists only of the objects compiled from
>>>> SPDK_ROOT_DIR/lib/env_dpdk.   With the ability to specify an alternate
>>>> environment, semantically, I had mistakenly assumed that the env library would
>>>> contain not only the SPDK implementation of the environment API, but also
>>>> everything on which it depended; i.e. for the default environment, the DPDK
>>>> objects.   But, that doesn't appear to be so.   Instead, when performing final
>>>> linking of each SPDK executable, not only is the SPDK environment library
>>>> specified, but so are the necessary DPDK libs on which it depends.   Variables
>>>> pulled in from the SPDK's environment makefile, env.mk, are used to specify
>>>> those DPDK libs along with additional, special linking flags.
>>>> 
>>>> We need to update the doc's Porting Guide to reflect these details, and I'd be
>>>> happy to volunteer to work on that effort. 
>>> 
>>> Please do!
>>> 
>>>> However, before embarking on that task, I'd like to pose the question:   are
>>>> there reasons why we couldn't or shouldn't produce the environment library to
>>>> consist of both the SPDK env implementation layer and the objects on which it
>>>> depends?  It would seem that would make final linking of executables a little
>>>> easier and reduce the complexity/effort of someone wanting to develop and use
>>>> an alternate environment.
>>> 
>>> How do you envision this playing with environment libraries that need to link
>>> against shared libraries, especially ones provided with the system itself? I'm
>>> not an expert in all of the available linker options, so maybe there is some way
>>> to embed enough information to properly link/load a shared library directly into
>>> the static library.
>>> 
>> 
>> This is an area in which I would greatly appreciate additional discussion, weighing the engineering cost/benefits of such flexibility. I, myself, could for example embrace the idea that for the sake of simplifying final linking that we propose the SPDK env implementation is built as a shared library which could be generated with a dependency on the DPDK (or its replaced equivalent) which also be shared lib(s).   Alternatively, I could also potentially be persuaded that we take an all static approach, though I hasten to add I can appreciate there may be stakeholders who strongly advocate/need shared libs.  I am aware of precedents where a package provided both static and shared versions of their libraries, and sometimes even in separate packages; i.e. a "static" package.   So, for greater flexibility, I could also envision a new SPDK configure option enabling the developer to specify static vs. dynamic, and perhaps with the options for the needed linking flags.  On the latt!
> er, I'm a
> ware per configure that it appears the build of the SPDK can/will honor shell environment variables for CFLAGS, CXXFLAGS, LDFLAGS and DESTDIR, but I don't know how challenging the use of those could be to override (or concatenate) those already figured out in the *mk/Makefile files.


I've only done a little experimenting in this area, too, but as you surmise it can get challenging indeed.

>> 
>> Out of curiosity, I retrieved the available DPDK packages from Ubuntu, and there are quite a few of them, including those with a static and dynamic dpdk library.   I haven't dug through all of the SPDK build options, but my initial pass would suggest it's currently built with only static env (and static DPDK) libs, at least for the default configuration.
>> 
>> Taking all of this a step further is the expectation that in the future we will provide two very different ways to build with the SPDK.   Today, we're checking out the SPDK repo and relying on the collection of makefiles therein to build the SPDK.   But, moving forward, we hope to produce an spdk-devel package(s) so that SPDK app developers could build against it/them.   Such a build environment there is outside our purview save for our need to provide the proper contents of those devel package(s).
>> 
>> In summary, I'm seeking, if possible, to simplify the build with respect to the env while keeping in mind the future with -devel package(s).   I'm very much hoping to get more input from others in the SPDK community on this topic, and am happy to volunteer on the implementation and documentation thereof once we can form a broader consensus.
>> 
>> thanks much,
>> 
> 
> I've thought about this some and I think we should make alternate environment
> library implementations shared libraries. It simplifies the linking and
> configuration considerably. If you'd like to take on the changes required to
> make this happen, please do so!


I'll take a swag at it.  Before that, though -- and related -- I intend to work on building individual SPDK shared-libs (https://github.com/spdk/spdk/issues/346 <https://github.com/spdk/spdk/issues/346>), an enhancement Daniel was going to field.   Should I create cards in Trello for these efforts, and if so which board -- "Things to Do"?   "Orchestration and Tooling Backlog"?

thanks,

--
Lance



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7238 bytes --]

Re: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 5014 bytes --]

On Mon, 2018-07-23 at 13:46 -0500, Lance Hartmann ORACLE wrote:
> > On Jul 20, 2018, at 5:17 PM, Walker, Benjamin <benjamin.walker(a)intel.com> wrote:
> > 
> > On Fri, 2018-07-20 at 16:34 -0500, Lance Hartmann ORACLE wrote:
> > > Another especially notable take-away for me was the realization that the
> > > environment library -- I'll refer to the default env for example here,
> > > libspdk_env_dpdk.a -- consists only of the objects compiled from
> > > SPDK_ROOT_DIR/lib/env_dpdk.   With the ability to specify an alternate
> > > environment, semantically, I had mistakenly assumed that the env library would
> > > contain not only the SPDK implementation of the environment API, but also
> > > everything on which it depended; i.e. for the default environment, the DPDK
> > > objects.   But, that doesn't appear to be so.   Instead, when performing final
> > > linking of each SPDK executable, not only is the SPDK environment library
> > > specified, but so are the necessary DPDK libs on which it depends.   Variables
> > > pulled in from the SPDK's environment makefile, env.mk, are used to specify
> > > those DPDK libs along with additional, special linking flags.
> > > 
> > > We need to update the doc's Porting Guide to reflect these details, and I'd be
> > > happy to volunteer to work on that effort. 
> > 
> > Please do!
> > 
> > > However, before embarking on that task, I'd like to pose the question:   are
> > > there reasons why we couldn't or shouldn't produce the environment library to
> > > consist of both the SPDK env implementation layer and the objects on which it
> > > depends?  It would seem that would make final linking of executables a little
> > > easier and reduce the complexity/effort of someone wanting to develop and use
> > > an alternate environment.
> > 
> > How do you envision this playing with environment libraries that need to link
> > against shared libraries, especially ones provided with the system itself? I'm
> > not an expert in all of the available linker options, so maybe there is some way
> > to embed enough information to properly link/load a shared library directly into
> > the static library.
> > 
> 
> This is an area in which I would greatly appreciate additional discussion, weighing the engineering cost/benefits of such flexibility. I, myself, could for example embrace the idea that for the sake of simplifying final linking that we propose the SPDK env implementation is built as a shared library which could be generated with a dependency on the DPDK (or its replaced equivalent) which also be shared lib(s).   Alternatively, I could also potentially be persuaded that we take an all static approach, though I hasten to add I can appreciate there may be stakeholders who strongly advocate/need shared libs.  I am aware of precedents where a package provided both static and shared versions of their libraries, and sometimes even in separate packages; i.e. a "static" package.   So, for greater flexibility, I could also envision a new SPDK configure option enabling the developer to specify static vs. dynamic, and perhaps with the options for the needed linking flags.  On the latter, I'm aware per configure that it appears the build of the SPDK can/will honor shell environment variables for CFLAGS, CXXFLAGS, LDFLAGS and DESTDIR, but I don't know how challenging the use of those could be to override (or concatenate) those already figured out in the *mk/Makefile files.
> 
> Out of curiosity, I retrieved the available DPDK packages from Ubuntu, and there are quite a few of them, including those with a static and dynamic dpdk library.   I haven't dug through all of the SPDK build options, but my initial pass would suggest it's currently built with only static env (and static DPDK) libs, at least for the default configuration.
> 
> Taking all of this a step further is the expectation that in the future we will provide two very different ways to build with the SPDK.   Today, we're checking out the SPDK repo and relying on the collection of makefiles therein to build the SPDK.   But, moving forward, we hope to produce an spdk-devel package(s) so that SPDK app developers could build against it/them.   Such a build environment there is outside our purview save for our need to provide the proper contents of those devel package(s).
> 
> In summary, I'm seeking, if possible, to simplify the build with respect to the env while keeping in mind the future with -devel package(s).   I'm very much hoping to get more input from others in the SPDK community on this topic, and am happy to volunteer on the implementation and documentation thereof once we can form a broader consensus.
> 
> thanks much,
> 

I've thought about this some and I think we should make alternate environment
library implementations shared libraries. It simplifies the linking and
configuration considerably. If you'd like to take on the changes required to
make this happen, please do so!

Thanks,
Ben

Re: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV

[Lance Hartmann ORACLE]

[-- Attachment #1: Type: text/plain, Size: 4633 bytes --]


> On Jul 20, 2018, at 5:17 PM, Walker, Benjamin <benjamin.walker(a)intel.com> wrote:
> 
> Hi Lance,
> 
> On Fri, 2018-07-20 at 16:34 -0500, Lance Hartmann ORACLE wrote:
>> Another especially notable take-away for me was the realization that the
>> environment library -- I'll refer to the default env for example here,
>> libspdk_env_dpdk.a -- consists only of the objects compiled from
>> SPDK_ROOT_DIR/lib/env_dpdk.   With the ability to specify an alternate
>> environment, semantically, I had mistakenly assumed that the env library would
>> contain not only the SPDK implementation of the environment API, but also
>> everything on which it depended; i.e. for the default environment, the DPDK
>> objects.   But, that doesn't appear to be so.   Instead, when performing final
>> linking of each SPDK executable, not only is the SPDK environment library
>> specified, but so are the necessary DPDK libs on which it depends.   Variables
>> pulled in from the SPDK's environment makefile, env.mk, are used to specify
>> those DPDK libs along with additional, special linking flags.
>> 
>> We need to update the doc's Porting Guide to reflect these details, and I'd be
>> happy to volunteer to work on that effort. 
> 
> Please do!
> 
>> However, before embarking on that task, I'd like to pose the question:   are
>> there reasons why we couldn't or shouldn't produce the environment library to
>> consist of both the SPDK env implementation layer and the objects on which it
>> depends?  It would seem that would make final linking of executables a little
>> easier and reduce the complexity/effort of someone wanting to develop and use
>> an alternate environment.
> 
> How do you envision this playing with environment libraries that need to link
> against shared libraries, especially ones provided with the system itself? I'm
> not an expert in all of the available linker options, so maybe there is some way
> to embed enough information to properly link/load a shared library directly into
> the static library.
> 


This is an area in which I would greatly appreciate additional discussion, weighing the engineering cost/benefits of such flexibility. I, myself, could for example embrace the idea that for the sake of simplifying final linking that we propose the SPDK env implementation is built as a shared library which could be generated with a dependency on the DPDK (or its replaced equivalent) which also be shared lib(s).   Alternatively, I could also potentially be persuaded that we take an all static approach, though I hasten to add I can appreciate there may be stakeholders who strongly advocate/need shared libs.  I am aware of precedents where a package provided both static and shared versions of their libraries, and sometimes even in separate packages; i.e. a "static" package.   So, for greater flexibility, I could also envision a new SPDK configure option enabling the developer to specify static vs. dynamic, and perhaps with the options for the needed linking flags.  On the latter, I'm aware per configure that it appears the build of the SPDK can/will honor shell environment variables for CFLAGS, CXXFLAGS, LDFLAGS and DESTDIR, but I don't know how challenging the use of those could be to override (or concatenate) those already figured out in the *mk/Makefile files.

Out of curiosity, I retrieved the available DPDK packages from Ubuntu, and there are quite a few of them, including those with a static and dynamic dpdk library.   I haven't dug through all of the SPDK build options, but my initial pass would suggest it's currently built with only static env (and static DPDK) libs, at least for the default configuration.

Taking all of this a step further is the expectation that in the future we will provide two very different ways to build with the SPDK.   Today, we're checking out the SPDK repo and relying on the collection of makefiles therein to build the SPDK.   But, moving forward, we hope to produce an spdk-devel package(s) so that SPDK app developers could build against it/them.   Such a build environment there is outside our purview save for our need to provide the proper contents of those devel package(s).

In summary, I'm seeking, if possible, to simplify the build with respect to the env while keeping in mind the future with -devel package(s).   I'm very much hoping to get more input from others in the SPDK community on this topic, and am happy to volunteer on the implementation and documentation thereof once we can form a broader consensus.

thanks much,

--
Lance Hartmann
lance.hartmann(a)oracle.com




[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7435 bytes --]

Re: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV

[Lance Hartmann ORACLE]

[-- Attachment #1: Type: text/plain, Size: 5199 bytes --]


> On Jul 18, 2018, at 11:35 PM, Luse, Paul E <paul.e.luse(a)intel.com> wrote:
> 
> Hi Lance,
>  
> I’m not sure when the last time someone actually tried this was – a few key folks are out this week so if you don’t hear something here you might try back or see if anyone familiar with actually doing this is on IRC over the next few days… would be great to fix anything here that’s busted though and really, really great to update the porting guide!


With much thanks to Ben yesterday,

https://review.gerrithub.io/#/c/spdk/spdk/+/419912/ <https://review.gerrithub.io/#/c/spdk/spdk/+/419912/>
https://review.gerrithub.io/#/c/spdk/spdk/+/419913/ <https://review.gerrithub.io/#/c/spdk/spdk/+/419913/>
https://review.gerrithub.io/#/c/spdk/spdk/+/419915/ <https://review.gerrithub.io/#/c/spdk/spdk/+/419915/>


it's possible again now to build with an alternate environment.   Digging through the SPDK build makefiles and working with Ben has raised my awareness of some of the details regarding alternate environment configuration and building.   The first item that becomes immediately apparent is that building with an alternate environment is not quite as simple as merely specifying '--with-env=/path/to/alt_env_directory'.   As currently implemented, the build of the SPDK requires that /path/to/alt_env_directory contains a makefile and it must be named 'env.mk', and therein specifically-named variables must be defined.  More on this later.

Another especially notable take-away for me was the realization that the environment library -- I'll refer to the default env for example here, libspdk_env_dpdk.a -- consists only of the objects compiled from SPDK_ROOT_DIR/lib/env_dpdk.   With the ability to specify an alternate environment, semantically, I had mistakenly assumed that the env library would contain not only the SPDK implementation of the environment API, but also everything on which it depended; i.e. for the default environment, the DPDK objects.   But, that doesn't appear to be so.   Instead, when performing final linking of each SPDK executable, not only is the SPDK environment library specified, but so are the necessary DPDK libs on which it depends.   Variables pulled in from the SPDK's environment makefile, env.mk, are used to specify those DPDK libs along with additional, special linking flags.

We need to update the doc's Porting Guide to reflect these details, and I'd be happy to volunteer to work on that effort. However, before embarking on that task, I'd like to pose the question:   are there reasons why we couldn't or shouldn't produce the environment library to consist of both the SPDK env implementation layer and the objects on which it depends?  It would seem that would make final linking of executables a little easier and reduce the complexity/effort of someone wanting to develop and use an alternate environment.


--
Lance Hartmann
lance.hartmann(a)oracle.com

> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lance Hartmann ORACLE
> Sent: Wednesday, July 18, 2018 5:07 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV
>  
>  
> I've been attempting to perform a build of the SPDK that relies on a different env_dpdk implementation, but am running into trouble.   The documentation's porting guide is a little sparse, but between that, a fragment from an email to the list:
>  
> [...]
> You can
> point SPDK at a different implementation of include/spdk/env.h through the
> configuration script:
>  
> ./configure --with-env=/path/to/your/lib.a
>  
> and some experimentation, I've not deduced exactly how this is supposed to work.
>  
> I grabbed a copy of a default built libspdk_env_dpdk.a and placed it in my ~/src/my_own_env_dpdk directory, and ran configure setting '--with-env=~/src/my_own_env_dpdk/libspdk_env_dpdk.a', but a build quickly fails because SPDK_ROOT_DIR/mk/spdk_common.mk attempts to do an include of an env.mk therein:
>  
>            SPDK_ROOT_DIR/mk/spdk_common.mk:170:  ~/src/my_own_env_dpdk/libspdk_env_dpdk.a/env.mk: Not a directory
>  
> The porting guide and the behavior of the build would indicate that CONFIG_ENV needs to be a directory.
>  
> Am I missing something fundamental in my configuration setup?   Or, perhaps, did this work a while back, but over time changes to the build process broke it?    I tried fudging with it further by doing things like copying the original SPDK_ROOT_DIR/lib/env_dpdk/env.mk to my own directory (and re-running the configure setting '--with-env' to just the directory, ~/src/my_own_env_dpdk), and kicking off the build.  That allows the build to run a lot further, but ultimately it fails later and I would suspect we wouldn't be imposing a requirement that one's own env implementation necessarily has to have an 'env.mk'.   Also, there seem to be other elements in the build process later on that don't consider CONFIG_ENV when you get to the linking stages.
>  
>  
> Thx,
> Lance
>  
>  
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 18687 bytes --]

Re: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2414 bytes --]

Hi Lance,

I'm not sure when the last time someone actually tried this was - a few key folks are out this week so if you don't hear something here you might try back or see if anyone familiar with actually doing this is on IRC over the next few days... would be great to fix anything here that's busted though and really, really great to update the porting guide!

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lance Hartmann ORACLE
Sent: Wednesday, July 18, 2018 5:07 PM
To: spdk(a)lists.01.org
Subject: [SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV


I've been attempting to perform a build of the SPDK that relies on a different env_dpdk implementation, but am running into trouble.   The documentation's porting guide is a little sparse, but between that, a fragment from an email to the list:

[...]
You can
point SPDK at a different implementation of include/spdk/env.h through the
configuration script:

./configure --with-env=/path/to/your/lib.a

and some experimentation, I've not deduced exactly how this is supposed to work.

I grabbed a copy of a default built libspdk_env_dpdk.a and placed it in my ~/src/my_own_env_dpdk directory, and ran configure setting '--with-env=~/src/my_own_env_dpdk/libspdk_env_dpdk.a', but a build quickly fails because SPDK_ROOT_DIR/mk/spdk_common.mk attempts to do an include of an env.mk therein:

           SPDK_ROOT_DIR/mk/spdk_common.mk:170:  ~/src/my_own_env_dpdk/libspdk_env_dpdk.a/env.mk: Not a directory

The porting guide and the behavior of the build would indicate that CONFIG_ENV needs to be a directory.

Am I missing something fundamental in my configuration setup?   Or, perhaps, did this work a while back, but over time changes to the build process broke it?    I tried fudging with it further by doing things like copying the original SPDK_ROOT_DIR/lib/env_dpdk/env.mk to my own directory (and re-running the configure setting '--with-env' to just the directory, ~/src/my_own_env_dpdk), and kicking off the build.  That allows the build to run a lot further, but ultimately it fails later and I would suspect we wouldn't be imposing a requirement that one's own env implementation necessarily has to have an 'env.mk'.   Also, there seem to be other elements in the build process later on that don't consider CONFIG_ENV when you get to the linking stages.


Thx,
Lance



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9394 bytes --]

[SPDK] Howto build with an external env, i.e. alternate CONFIG_ENV

[Lance Hartmann ORACLE]

[-- Attachment #1: Type: text/plain, Size: 1796 bytes --]


I've been attempting to perform a build of the SPDK that relies on a different env_dpdk implementation, but am running into trouble.   The documentation's porting guide is a little sparse, but between that, a fragment from an email to the list:

[...]
You can
point SPDK at a different implementation of include/spdk/env.h through the
configuration script:

./configure --with-env=/path/to/your/lib.a

and some experimentation, I've not deduced exactly how this is supposed to work.

I grabbed a copy of a default built libspdk_env_dpdk.a and placed it in my ~/src/my_own_env_dpdk directory, and ran configure setting '--with-env=~/src/my_own_env_dpdk/libspdk_env_dpdk.a', but a build quickly fails because SPDK_ROOT_DIR/mk/spdk_common.mk attempts to do an include of an env.mk therein:

	SPDK_ROOT_DIR/mk/spdk_common.mk:170:  ~/src/my_own_env_dpdk/libspdk_env_dpdk.a/env.mk: Not a directory

The porting guide and the behavior of the build would indicate that CONFIG_ENV needs to be a directory.

Am I missing something fundamental in my configuration setup?   Or, perhaps, did this work a while back, but over time changes to the build process broke it?    I tried fudging with it further by doing things like copying the original SPDK_ROOT_DIR/lib/env_dpdk/env.mk to my own directory (and re-running the configure setting '--with-env' to just the directory, ~/src/my_own_env_dpdk), and kicking off the build.  That allows the build to run a lot further, but ultimately it fails later and I would suspect we wouldn't be imposing a requirement that one's own env implementation necessarily has to have an 'env.mk'.   Also, there seem to be other elements in the build process later on that don't consider CONFIG_ENV when you get to the linking stages.


Thx,
Lance




[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3047 bytes --]