Re: [SPDK] Building spdk on CentOS6

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 11637 bytes --]

On Tue, 2018-05-08 at 14:34 +0000, Shahar Salzman wrote:
> Got it.
> 
> I was tracking the dpdk project, and focused on the branch that Daniel told me
> to work on.
> So would the correct process be to push the patch to the spdk/dpdk gerrithub
> and discuss the commit, or the other way around?

Yes - go ahead and push to spdk/dpdk on GerritHub so we can all review it. The
SPDK project has never written out a policy on which versions of DPDK are
supported for any given SPDK release. The code is always tested using the latest
released DPDK, and then attempts to support older versions of DPDK are done in
an ad hoc manner. Similarly, the project never explicitly states which operating
systems are supported. Instead, we indicate which operating systems the code is
regularly tested on (Fedora N and N-1, Ubuntu latest, Ubuntu LTS, CentOS 7,
FreeBSD 11).

I see that CentOS 6 is no longer receiving full updates and is in maintenance
only mode. It will stop receiving even maintenance updates in 2020. I think that
is indicative of the level of support SPDK should provide here.

Given the above, if this patch can be applied to the DPDK fork and doesn't have
any additional impact, I'm not opposed to applying it. That wouldn't, in my
mind, mean that SPDK is committed to supporting CentOS 6 indefinitely. But
certainly that support could continue while it is relatively easy and
convenient.

> 
> From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Luse, Paul E <paul.e.luse@
> intel.com>
> Sent: Tuesday, May 8, 2018 5:14:10 PM
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] Building spdk on CentOS6
>  
> Sure, I understand that.  SPDK maintains a fork of dpdk in github at https://g
> ithub.com/spdk/dpdk for the purpose of temporarily putting things in that we
> need that can’t make it upstream yet for whatever reason – this is the
> submodule in the SPDK repo.  I’m not saying that it makes sense for your patch
> to land there permanently but it does make sense to talk about it.
>  
> No problem on not being on the call, we’ll get some input from the maintainers
> and others I’m sure…
>  
> Thx
> Paul
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar Salzman
> Sent: Tuesday, May 8, 2018 7:04 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Building spdk on CentOS6
>  
> Hey Paul,
>  
> The problem is that the patch is in dpdk and not in spdk, and this is the
> reason I offered to support it.
> The patch (bellow) is only a few lines long modifying the deprecated attribute
> definition, a modification to the config, and an extra compilation flag.
> All of the above are required in order to build dpdk 18.01 on CentOS6, there
> are no problems building spdk on CentOS6.
>  
> I understood that you are already applying minor patches to dpdk, so question
> is, whether this patch can be added?
>  
> Unfortunately due to some personal issues, I am unable to participate in
> today's meeting.
>  
> Thanks,
> Shahar
> From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Luse, Paul E <paul.e.luse@
> intel.com>
> Sent: Tuesday, May 8, 2018 4:25:29 PM
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] Building spdk on CentOS6
>  
> Hi Shahar,
>  
> Couple of quick things:
>  
> * thanks for putting all this time and effort into trying to make this work!
> * in the community model, patches aren’t accepted based on one individual’s
> ability/desire to support them so appreciate the offer but in general any
> patch that is accepted becomes the responsibly of the community (mostly the
> maintainers) to support it long term.  Otherwise we’d have what I like to call
> “the flea market model” where there’s a bunch of separate individuals
> supporting their own things with very little cohesiveness across everything J
>  
> So, wrt next steps, if the patch isn’t a tremendous amount of effort I would
> suggest you go ahead and submit it. That’s the best way to get everyone on the
> same page wrt exactly what we’re talking about.  If it is some significant
> effort then, as Pawel states, you can call into a community meeting (see email
> I sent out last night) or you can explain more details on this list.
>  
> Thanks again!!
> Paul
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wodkowski, PawelX
> Sent: Tuesday, May 8, 2018 5:57 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Building spdk on CentOS6
>  
> I think someone will answer you on this mailing list but there is community
> meeting today too. You can raise this if you wish.
>  
> Pawel
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar Salzman
> Sent: Tuesday, May 8, 2018 2:23 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Building spdk on CentOS6
>  
> Hi,
>  
> I attempted to push the gcc support patch to dpdk, and got rejected since they
> do not wish to support CentOS6.
> Would it be possible to manually add the patch to the supported dpdk version
> (currently 18.01).
> I would be happy to support CentOS6 issues.
>  
> Thanks,
> Shahar
>  
> From: Shahar Salzman
> Sent: Tuesday, April 24, 2018 3:40:51 PM
> To: Storage Performance Development Kit
> Cc: Ilan Steinberg; Ido Benda; Yael Shavit
> Subject: Re: Building spdk on CentOS6
>  
> OK, got everything to work.
>  
> For DPDK, I had to to do the following:
> add a patch to make the new __rte_experimental attribute to work
> remove a bunch of modules via config which had some gnarly compilation errors,
> and are not needed for spdk
> use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict aliasing
> errors.
>  
> I tried spdk 18.01, and ran into some issues with the shared library build. As
> I saw that Daniel had already dealt with them, I tried cherry picking, but
> eventually used the master branch.
> I got these type of warnings:
>  
> /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by
> /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp-
> gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link)
> /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by
> /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp-
> gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link)
>  
> I added the following patch bellow to make them go away.
>  
> I'll approach the dpdk community on the mailing list in order to get the
> __rte_experimental stuff to work.
> My final configuration is:
> CenrOS6.4
> gcc 4.4.7-17
> dpdk v18.02
> spdk master
>  
> What are the next steps?
>  
> Shahar
>  
> P.S. Here is the patch to remove the warnings above.
> commit d4ef744de1cc4928972b7042d1c90aff12db7867
> Author: shahar salzman <shahar.salzman(a)kaminario.com>
> Date:   Tue Apr 24 14:31:52 2018 +0300
>  
>     lib/env_dpdk: add required libraries
>  
> diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk
> index 450043c..b46bfed 100644
> --- a/lib/env_dpdk/env.mk
> +++ b/lib/env_dpdk/env.mk
> @@ -68,6 +68,18 @@ ifneq ($(wildcard $(DPDK_ABS_DIR)/lib/librte_malloc.*),)
>  DPDK_LIB_LIST += rte_malloc
>  endif
>  
> +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*))
> +DPDK_LIB_LIST += rte_mbuf
> +endif
> +
> +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*))
> +DPDK_LIB_LIST += rte_net
> +endif
> +
> +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*))
> +DPDK_LIB_LIST += rte_ethdev
> +endif
> +
>  # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these
> libraries conditionally
>  # based on their existence to maintain backward compatibility.
>  ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*))
>  
>  
>  
> From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Shahar Salzman <shahar.sal
> zman(a)kaminario.com>
> Sent: Tuesday, April 24, 2018 9:57 AM
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] Building spdk on CentOS6
>  
> Great, and thanks!
>  
> I have been running into issues compiling dpdk 17.11 on CentOS6 (there seems
> to be a gcc issue that does not exist in other versions), I'll focus my
> efforts on dpdk 18.02.
> From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Verkamp, Daniel <daniel.ve
> rkamp(a)intel.com>
> Sent: Monday, April 23, 2018 9:40:36 PM
> To: Storage Performance Development Kit
> Subject: Re: [SPDK] Building spdk on CentOS6
>  
> Hi Shahar,
>  
> I’ve pushed a patch for the SPDK master branch that should allow it to compile
> with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/
>  
> Gerrit Code Review
>  
> review.gerrithub.io
>  
> Keep in touch. Copyright © 2017 | GerritForge Ltd. info(a)gerritforge.com
> www.gerritforge.com
>  
>  
> However, DPDK 16.07 is no longer supported upstream; I highly recommend that
> you update to a newer version. SPDK is currently tested with DPDK 18.02.
>  
> The VFIO issue you mention has also been fixed on SPDK master, and it will be
> part of the SPDK v18.04 release.
>  
> Thanks,
> -- Daniel
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Shahar Salzman
> Sent: Sunday, April 22, 2018 6:41 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Building spdk on CentOS6
>  
> Hi,
>  
> Finally got to looking at support of spdk build on CentOS6, things look good,
> except for one issue.
> spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow
> compilation) and some minor patches (mainly some memory configuration stuff),
> kernel is a patched 4.9.6.
>  
> build succeeded except for the usage of the dpdk function pci_vfio_is_enabled.
> I had to apply the patch bellow, removing the usage of this function and then
> compilation completed without any issues.
>  
> It seems that I am missing some sort of dpdk configuration as I see that the
> function is built, but not packaged into the generated archive.
>  
> I went back to square one and ran the instructions in http://www.spdk.io/doc/g
> etting_started.html, but I see no mention of dpdk there. Actually the
> ./configure requires it.
>  
> My next step is to use a more recent dpdk, but shouldn't this work with my
> version? Am I missing some dpdk configuration?
> BTW, as we are not using vhost, on our 17.07 version we simply use
> CONFIG_VHOST=n in order to skip this, but I would be happier if we used a
> better solution.
>  
> Shahar
>  
> P.S. Here is the patch to remove use of this function:
> diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c
> index 92aa256..f38929f 100644
> --- a/lib/env_dpdk/vtophys.c
> +++ b/lib/env_dpdk/vtophys.c
> @@ -53,8 +53,10 @@
>  #define SPDK_VFIO_ENABLED 1
>  #include <linux/vfio.h>
>  
> +#if 0
>  /* Internal DPDK function forward declaration */
>  int pci_vfio_is_enabled(void);
> +#endif
>  
>  struct spdk_vfio_dma_map {
>         struct vfio_iommu_type1_dma_map map;
> @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void)
>         DIR *dir;
>         struct dirent *d;
>  
> +#if 0
>         if (!pci_vfio_is_enabled()) {
>                 return;
>         }
> +#endif
>  
>         dir = opendir("/proc/self/fd");
>         if (!dir) {
>  
>  
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk