Re: [SPDK] Shared library build

* Re: [SPDK] Shared library build
@ 2017-09-13 11:03 Jonas Pfefferle1
  0 siblings, 0 replies; 10+ messages in thread
From: Jonas Pfefferle1 @ 2017-09-13 11:03 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 12383 bytes --]

Hi Daniel,

Great. I guess this will be included in the 17.10 release?
I believe most people don't assume a stable API at this point but I
understand that this is a problem. Maybe marking functions as stable as
soon as we are sure that they don't change anymore could be a start and for
the other functions a big warning that they might break compatibility
between the so versions.

Thanks,
Jonas

From:	"Verkamp, Daniel" <daniel.verkamp(a)intel.com>
To:	Storage Performance Development Kit <spdk(a)lists.01.org>
Date:	09/09/2017 01:08 AM
Subject:	Re: [SPDK] Shared library build
Sent by:	"SPDK" <spdk-bounces(a)lists.01.org>

Hi Jonas,

We should definitely add shared library build to the backlog.  However,
there’s a few reasons we don’t support it today:
      -        The APIs we provide are not stable – we are still rapidly
      changing and improving them currently.  Once things settle down, we
      can start to think about API and ABI compatibility, but currently
      it’s not very useful to have a shared library without the ability to
      upgrade it to a newer version with a compatible soname.
      -        Our build system doesn’t currently consider dependencies
      between libraries, since it doesn’t matter for static libs –
      everything is built separately and only linked at the final link
      step, and all of the programs that need to be linked to libs are in
      separate top-level dirs (app, examples, test) that have a Makefile
      dependency on the lib directory.  If we want to support a shared
      library build, we’ll have to add some more dependency information to
      the build system so that the libraries in ‘lib’ can be built in the
      correct order so they can refer to each other.

That said, these challenges can be overcome, and we’ll be working toward
supporting a shared lib build in the future.

Regarding your question about the NVMe-oF target library’s dependency on
the event framework, you are correct that they are coupled together in SPDK
17.07, but we have been working to decouple them, and libspdk_nvmf should
be usable now without linking to the event framework.  The cross-thread
communication is now provided via the io_channel.h abstractions (in the
case of our example app/nvmf_tgt, this still uses the event framework, but
you’re now able to plug in your own implementation).

Thanks,
-- Daniel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle1
Sent: Friday, September 1, 2017 8:28 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Shared library build

Hi Paul,

I will put something on Trello. Correct me if I'm wrong but in 17.07 I
cannot create a new nvmf subsystem without calling spdk_reactors_init first
because subsystem_start sets up a poller which needs reactor. This function
is called by spdk_app_start and is not exported through the public API (in
spdk_internal/event.h).

Thanks,
Jonas

Inactive hide details for "Luse, Paul E" ---09/01/2017 04:59:28 PM---Hi
Jonas, We were *just* talking about that like 2 days ag"Luse, Paul E"
---09/01/2017 04:59:28 PM---Hi Jonas, We were *just* talking about that
like 2 days ago I think. I don’t believe anyone has put

From: "Luse, Paul E" <paul.e.luse(a)intel.com>
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: 09/01/2017 04:59 PM
Subject: Re: [SPDK] Shared library build
Sent by: "SPDK" <spdk-bounces(a)lists.01.org>

Hi Jonas,

We were *just* talking about that like 2 days ago I think. I don’t believe
anyone has put anything up on Trello for it yet but I will, or feel free if
you’d like, to get it some more visibility. The idea, that is, of a super
lightweight env library. We were thinking more along the lines for
development use and as a way to test the env.h abstraction but, hey,
anything’s possible!

Thx
Paul

PS: you’re probably well aware of this but FYI for others, spdk_app_start()
is actually part of the SPDK app framework so calling public SPDK functions
directly is totally a valid usage model. The framework is provided for
those who wish to use it (maybe they don’t have something already in place)
and to help build the suite of apps and examples in the repo, well, and for
testing as well. Daniel did a preso on the topic at the summit back in Apr,
deck is here:
https://ci.spdk.io/download/events/2017-summit/05_-_Day_1_-_Verkamp_-_Under_the_Hood_v10.pdf

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle1
Sent: Friday, September 1, 2017 1:11 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Shared library build

I agree the biggest hurdle is the DPDK dependency. It would be nice to have
a lightweight env library (e.g. by removing the necessary parts from DPDK
and making it available as a standalone library for both SPDK and DPDK to
use). However as you pointed out there are still some things in SPDK that
don't feel like a library, i.e. spdlk_env_init exiting with exit(-1)
instead of returning an error.
Great to hear you are working on making the NVMf code more library
friendly. We currently work around this issue by calling SPDK internal
functions to setup a target without using spdk_app_start etc.

Thanks,
Jonas

Inactive hide details for "Walker, Benjamin" ---08/31/2017 08:31:11 PM---I
think a lot of us have an intuitive sense of what yo"Walker, Benjamin"
---08/31/2017 08:31:11 PM---I think a lot of us have an intuitive sense of
what you mean by "as a library" but I just wanted to

From: "Walker, Benjamin" <benjamin.walker(a)intel.com>
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: 08/31/2017 08:31 PM
Subject: Re: [SPDK] Shared library build
Sent by: "SPDK" <spdk-bounces(a)lists.01.org>

I think a lot of us have an intuitive sense of what you mean by "as a
library" but I just wanted to take the opportunity to clarify a bit.

DPDK is what I'd call a "framework" - it owns the main loop of the
application, it manages all memory, it spawns its own threads, etc. To use
code from DPDK you have to buy in to the framework, generally.

SPDK, on the contrary, is mostly a collection of libraries. These libraries
integrate into your existing application as opposed to requiring your
application to buy in. There are a couple of exceptions - notably lib/event
which is a framework we use for writing all of our example apps, but not
used in any libraries.*

SPDK does, by default, depend on DPDK though. So how can SPDK be a
collection of libraries when it depends on DPDK? All DPDK usage is wrapped
by the SPDK environment abstraction library (lib/env_dpdk and
include/spdk/env.h), and that abstraction doesn't require any of the
"framework-y" aspects. Further, to remove the DPDK dependency entirely,
just reimplement the env.h header as a static library and pass the location
of the library to our configure script.

In general, I don't see any problem with building SPDK as a set of shared
libraries. Someone just needs to go add the option to our build system.

For lib/nvmf specifically, the line between the app (app/nvmf_tgt) and the
library is not well fleshed out in 17.07. I've recently done a lot of work
on lib/nvmf though, so that line is starting to get much clearer. I think
with a few more cleanups it would be a lot easier to use lib/nvmf as a
shared library in an application. That's certainly our long term direction.

*There may be a couple of spots that still use lib/event from libraries but
we're working to remove them.

-------- Original message --------
From: "Luse, Paul E" <paul.e.luse(a)intel.com>
Date: 8/31/17 10:14 AM (GMT-07:00)
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Shared library build

That’s way cool to see DiSNI out there!

Pankaj, I’m sure many are curious as to the challenges you reference so
please feel free to go a little deeper there and most definitely if you
believe any modifications you’ve made in this might be beneficial to others
by all means throw a patch out there. Is this something you can share?

Jonas, wrt your question yeah I tend to agree. There are a lot of moving
parts in the SPDK project and it may make sense in some cases and not in
others. I know there are some others out there that have done some work
relevant to the conversation but I’m not super familiar with it so I’ll let
one of them chime in…

Thx
Paul

PS: reminder that the SPDK community is also on freenode, channel #spdk

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Jonas Pfefferle1
Sent: Thursday, August 31, 2017 5:23 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Shared library build

Hi Pankaj,

Technically we already use SPDK as a library in the disni project I'm
working on (https://github.com/zrlio/disni) but I think this is orthogonal
to being able to build it as a shared library. E.g. DPDK has the option to
build shared libraries but is also not really meant to be used as a
library. Whether you can/should use SPDK/DPDK as a library is a whole
different discussion.

Regards,
Jonas

Inactive hide details for "Pankaj P Deshpande1" ---08/31/2017 01:49:18
PM---  Jonas,   There is no  target : library  option in"Pankaj P
Deshpande1" ---08/31/2017 01:49:18 PM--- Jonas, There is no target :
library option in SPDK. However, we have modified SPDK to make i

From: "Pankaj P Deshpande1" <Pankaj.Deshpande(a)in.ibm.com>
To: spdk(a)lists.01.org
Cc: spdk(a)lists.01.org
Date: 08/31/2017 01:49 PM
Subject: Re: [SPDK] Shared library build
Sent by: "SPDK" <spdk-bounces(a)lists.01.org>

Jonas,

There is no target:library option in SPDK. However, we have modified SPDK
to make it a library in one of the POCs I am part of. We are aware of many
challenges involved in using SPDK as nvmf library and few solutions.

BTW, I too work for IBM, so we can syncup on this soon along with Ioannis
K :-).

Pankaj Deshpande

----- Original message -----
From: "Jonas Pfefferle1" <JPF(a)zurich.ibm.com>
Sent by: "SPDK" <spdk-bounces(a)lists.01.org>
To: spdk(a)lists.01.org
Cc:
Subject: [SPDK] Shared library build
Date: Thu, Aug 31, 2017 4:53 PM

Hi,

Is there a way to build SPDK as a shared library? If no, are there any
plans to support this?

Regards,
Jonas

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_spdk&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=OJny1i-VafTz20CwLEKEtwx6EAnCNqX0f_-eM5rVa8U&m=WM-_uIuqFhlGD5a3MEzoQqHFhfFpYeyVWdw4bdrNctU&s=oXt1qoGCavTP6XszW47jglecRLR8AtKy5XvS130fCmU&e=

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_spdk&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-pXjigIjRW0&m=WP7a7j8f_8MWf1t_3VPJ6li9MS_bmoXordH_pRoCQoE&s=t0vYbeGNlRWwwgcxkciIr3RJzqicmDuxQH6CXIuw9zE&e=

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_spdk&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-pXjigIjRW0&m=92HqwnID4elXWdQVN0-nTqb8nVKZeQ3AL-z_rzBG5gA&s=33hHN6svTzbyUfp5_VM66bSrk1By9nbTCin_2-U5Joc&e=

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_spdk&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-pXjigIjRW0&m=7786iujuvudtg1gbAhg9k78yNAvBnDom-XXKF7pXt68&s=lUAdLnXpd38ZzQnbU2T1rU7T0lKMmUIqgpIpsId7VZQ&e=

 _______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.01.org_mailman_listinfo_spdk&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-pXjigIjRW0&m=AOA3hENpOkfa2SRd6PI0J09RulRwos-3-Lie5HyZE28&s=4P7rJpFWVIDcrWN1FNPSNNsvVybzsF5chTZoFNxl5DA&e=

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 25604 bytes --]

[-- Attachment #3: graycol.gif --]
[-- Type: image/gif, Size: 105 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread