Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture

* Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture
@ 2018-06-17  0:20 Zhipeng Huang
  0 siblings, 0 replies; 15+ messages in thread
From: Zhipeng Huang @ 2018-06-17  0:20 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 13376 bytes --]

Thanks Paul,

Will there is no extra features per se at the moment, but we will gradually
rolls that out hopefully. The current spdk driver code could be found at
https://github.com/openstack/cyborg/blob/master/cyborg/accelerator/drivers/spdk/nvmf/nvmf.py

Well the purpose of the discussion is actually quite simple, we suggest to
add Cyborg as another project like Cinder which supports spdk in the SPDK
architecture figure :) I think our work in Cyborg is good for SPDK's
ecosystem and would be great to acknowledge that :)

Sorry for the many lengthy emails :)

On Sat, Jun 16, 2018 at 11:15 PM Luse, Paul E <paul.e.luse(a)intel.com> wrote:

> Thanks for the information! Are there currently any Cyborg drivers (is
> that the right term?) that implement NVMeOF? It would be great to take a
> look at one to help clear things up a bit – what I’m still not seeing is
> exactly what “extra capabilities” you mention below that could be
> exposed/exploited via Cyborg & SPDK vs Cinder & SPDK.
>
>
>
> Also, assuming that nothing else is required from the SPDK
> codebase/community, is the purpose of this email chain just to
> inform/educate and help enlist the SPDK community in promoting its use in
> Cyborg?
>
>
>
> Note that I’m on holiday from now until July 2 and starting Mon afternoon
> will not have access to email until I get back so when you don’t get any
> kind of response from me, that’s why J Hopefully others will carry the
> conversation forward as I know others are interested as well and fully
> appreciate all of Helloway’s interest in SPDK thus far.  I’ll definitely
> touch base when I return, thanks again!
>
>
>
> -Paul
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Zhipeng
> Huang
> *Sent:* Friday, June 15, 2018 6:27 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture
>
>
>
> Hi Maciek and Paul,
>
>
>
> I'm the current project lead of Cyborg so let me try to answer your
> questions, maybe hellowway did not explain well before.
>
>
>
> Cyborg[0] is a general management framework for accelerators (GPU. FPGA,
> ASIC, NVMe/NOF, ...). We actually worked with Moshe's team in the project's
> early days :) . The support of spdk driver has already landed in cyborg
> (although still very premature).
>
>
>
> It is important to notice that Cyborg does not provide volume management
> which is the job of Cinder. Cyborg provide management on NVMe/NVMe Over
> Fabric SSDs, meaning leveraging information that could be attained via the
> Bus (and the NVMe/NOF protocol). A similar comparison could be found in
> Cyborg management of FPGA/GPU. This could enable many great features
> developed in NVMe protocol could be exposed via Cyborg to support more
> granulized scheduling if the user want, instead of just being treated as a
> block device without difference to traditional SSDs.
>
>
>
> Therefore Cyborg is working alongside of Nova and Cinder. An example of
> how Cyborg work with Nova on NVMe devices could be found in [1] starting
> 27:13, which is also proposed from Intel developers. In essence, Cyborg
> helps Nova get knowledge of NVMe SSDs for its various extra capabilities
> compared to normal SSD (i.e, as an accelerator), Nova scheduler then will
> select a node with such capability if the user desires to spawn a VM, and
> then Cinder will just do its job on volume management.
>
>
>
> So on this note, Cyborg is working with Nova and Cinder, the relationship
> is complimentary. Cyborg will interact with Nova through Placement, there
> is no need at the moment for Cyborg to interact with Cinder. It could still
> work without Cyborg in the picture for sure, but NVMe SSD will be seen just
> as a normal block device by Nova and Cinder, and no advanced scheduling
> could be performed.
>
>
>
> Re Paul's question:
>
> - Just a good Cyborg SPDK driver could make it work in OpenStack
>
> - No additional requirement on SPDK community itself
>
> - No additional tweaks in SPDK needed specifically for Cyborg to work with
> it.
>
>
>
> OpenSDS is just another option, since it supports capability report
> functionality, it could gather information from Cyborg to make more
> granulized scheduling, unlike Cinder which does not get input from Cyborg
> and just perform regular volume management. There is no good or bad to
> compare two solutions. It depends on user's requirement.
>
>
>
> I hope this writing could clear things up :)
>
>
>
> [0] https://wiki.openstack.org/wiki/Cyborg
>
> [1] https://www.youtube.com/watch?v=q84Q-6FSXts
>
>
>
>
>
> On Fri, Jun 15, 2018 at 10:28 PM Szwed, Maciej <maciej.szwed(a)intel.com>
> wrote:
>
> Hi Helloway,
>
> I’ve been working on Cinder drivers (volume and target) for SPDK. I’m not
> familiar with Cyborg. Does Cyborg have capabilities to manage volumes? How
> does it interact with Nova – how does it provide storage to compute nodes?
>
>
>
> Thanks,
>
> Maciek
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Luse, Paul
> E
> *Sent:* Thursday, June 14, 2018 5:27 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture
>
>
>
> Hi Helloway,
>
>
>
> No problem.  Cinder is the block storage provisioning project for
> Openstack. A VM needing block storage interacts with Cinder to request what
> it needs and storage providers that have written provider (aka drivers) for
> Cinders are matched up based on requests. Here’s the overview from the
> project:
> https://docs.openstack.org/newton/install-guide-ubuntu/common/get-started-block-storage.html
> and this is by far the most common way that I’m aware of to enable a new
> block storage application in OpenStack.
>
>
>
> I haven’t worked in that world in a few years such that Cyborg and OpenSDS
> have been introduced since I was active but as far as I know Cinder is
> still the best place to start introducing SPDK based block storage into the
> Openstack cloud. I do have some OpenSDS contacts, and still some friends
> who work on Cinder.  Let me ask around a little, note that I will be out of
> the office for 2 weeks of “disconnected” vacation after this Friday but
> I’ll try and get a bit more info before then and get back to ya.
>
>
>
> Anyone else out there feel free to chime in if you have more info on
> Cyborg+OpenSDS vs Cinder as it pertains to this discussion. Once we figure
> that out, the questions below are still very relevant for next steps:
>
>
>
> - what else is required to be pushed into OpenStack for this to work
>
> - is anything required in the SPDK repo for this to work
>
> - how will the necessary SPDK components be associated with the VM in
> question and subsequently configured
>
>
>
> Thanks!
>
> Paul
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *helloway
> *Sent:* Wednesday, June 13, 2018 7:47 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Cc:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture
>
>
>
> Hi Paul,
>
>
>
> I am sorry for that I'm not an expert on Cinder, I‘ll try to tell you what
> I think. If I was wrong, please correct me. From my perspective, Cinder
> cares more about the capacity of the pool whose pool capabilities reporting
> interface is fixed, whereas, in addition to the capacity, Cyborg also cares
> about the fine-grained accelerator capabilities (e.g. iops, queue,
> etc.). These capabilities, reported  from the Cyborg, can be dynamic
> configured and handled through the OpenSDS’ profile. For this reason, it
> provides a more flexible and simple configuration which can be called
> conveniently.
>
>
>
> Thx,
>
> Helloway
>
>
>
> On 06/13/2018 22:46，Luse, Paul E<paul.e.luse(a)intel.com>
> <paul.e.luse(a)intel.com> wrote：
>
> Hi Helloway,
>
>
>
> That’s a great start but I still have the same open questions below, maybe
> you can try and address those directly? Also, below is the link for adding
> SPDK based NVMeOF as a Cinder plug-in.  In addition to the question below
> can you please explain for everyone how you see the approach of using
> Cyborg and OpenSDS compares with the seemingly simpler approach of
> providing a Cinder plug-in?
>
>
>
> https://review.openstack.org/#/c/564229/
>
>
>
> Thanks!!
>
> Paul
>
>
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *helloway
> *Sent:* Tuesday, June 12, 2018 5:22 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Cc:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture
>
>
>
> Hi Paul,
>
> Thank you for your response, I have submitted a trello [0] titled
>  “Integrate OpenStack/Cyborg into SPDK Architecture”. I am trying to answer
> your questions from this trello, do I make sense? I really hope to receive
> your feedback.
>
>
>
> [0]
> https://trello.com/c/QfSAkLSS/121-integrate-openstack-cyborg-into-spdk-architecture
>
>
>
>
> Thx,
>
> Helloway
>
> On 06/12/2018 08:16，Luse, Paul E<paul.e.luse(a)intel.com>
> <paul.e.luse(a)intel.com> wrote：
>
> Hi Helloway,
>
>
>
> I was actually just wondering what had happened with this.  Looking at the
> OpenStack patch it looks like it’s close to landing. Somewhere out there we
> have a Cinder driver that’s also getting fairly close I believe so for sure
> integration with OpenStack is interesting to many in the community.
>
>
>
> Would you be able to summarize more specifically how your patch would work
> once it lands? I of course see your high level description below that some
> questions that I have, I’m assuming others as well, include:
>
>
>
> - what else is required to be pushed into OpenStack for this to work
>
> - is anything required in the SPDK repo for this to work
>
> - how will the necessary SPDK components be associated with the VM in
> question and subsequently configured
>
>
>
> Thanks for continuing to work on this!
>
>
>
> -Paul
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *helloway
> *Sent:* Monday, June 11, 2018 1:35 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* [SPDK] Integrate OpenStack/Cyborg into SPDK Architecture
>
>
>
> Hi Jim and all,
>
>
>
> Do you know OpenStack/Cyborg? It is OpenStack acceleration service which
> provides a management framework for accelerator devices (e.g. FPGA, GPU,
> NVMe SSD). There is a strong demand for OpenStack to support hardware
> accelerated devices in a dynamic model(mentioned in OpenStack Summit
> Vancouver 2018 [0]).
>
>
>
> For this reason, we can use Cyborg to interactive with nvmf_tgt to realize
> the management of the user space accelerator NVMe SSD device, which can
> badly promote the efficiency. It is worth mentioning that the
> Cyborg_SPDK_Driver I summitted has been merged into the OpenStack version Q
> [1]. The driver can report the detailed information of the device to the
> Cyborg agent. When user requests a vm with a user space NVMe SSD, Cyborg
> agent will update the Nova/Placement inventory on available NVMe devices.
> This is a complete process to describe the connection of Cyborg and SPDK.
>
>
>
> I wonder whether you guys are interested in integrating OpenStack/Cyborg
> into SPDK architecture? Do I make sense? Please let me know what your
> thoughts.
>
>
>
> [0]
> https://www.openstack.org/videos/vancouver-2018/optimized-hpcai-cloud-with-openstack-acceleration-service-and-composable-hardware
>
> [1]https://review.openstack.org/#/c/538164/
>
>
>
>
>
> Thx,
>
> Helloway
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
> --
>
> Zhipeng (Howard) Huang
>
>
>
> Standard Engineer
>
> IT Standard & Patent/IT Product Line
>
> Huawei Technologies Co,. Ltd
>
> Email: huangzhipeng(a)huawei.com
>
> Office: Huawei Industrial Base, Longgang, Shenzhen
>
>
>
> (Previous)
>
> Research Assistant
>
> Mobile Ad-Hoc Network Lab, Calit2
>
> University of California, Irvine
>
> Email: zhipengh(a)uci.edu
>
> Office: Calit2 Building Room 2402
>
>
>
> OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

-- 
Zhipeng (Howard) Huang

Standard Engineer
IT Standard & Patent/IT Product Line
Huawei Technologies Co,. Ltd
Email: huangzhipeng(a)huawei.com
Office: Huawei Industrial Base, Longgang, Shenzhen

(Previous)
Research Assistant
Mobile Ad-Hoc Network Lab, Calit2
University of California, Irvine
Email: zhipengh(a)uci.edu
Office: Calit2 Building Room 2402

OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 34471 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread