From mboxrd@z Thu Jan  1 00:00:00 1970
Content-Type: multipart/mixed; boundary="===============4314877694713167070=="
MIME-Version: 1.0
From: Gyan Prakash <gyapra2016 at gmail.com>
Subject: Re: [SPDK] Building spdk on CentOS6
Date: Fri, 24 Aug 2018 15:00:20 -0700
Message-ID: <CADa+-y+wscorE_WsvbxG=eP7euOrF-2fHLb1zMVDb8dzCECE4A@mail.gmail.com>
In-Reply-To: AM3PR04MB3704A98C25BEB64312BEFAD899D0@AM3PR04MB370.eurprd04.prod.outlook.com
List-ID: <spdk@lists.01.org>
To: spdk@lists.01.org

--===============4314877694713167070==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

Hi all,

I am trying to build spdk-18.07 with dpdk-18.05 on Centos 7.4 system with
kernel 4.13.9.

make DPDK_DIR=3D./dpdk-18.05/x86_64-native-linuxapp-gcc CONFIG_RDMA=3Dy

and seeing following compilation failure. Can someone please tell me what I
am missing there?

CC lib/vhost/vhost.o
In file included from vhost_internal.h:39:0,
                 from vhost.c:43:
rte_vhost/rte_vhost.h:47:24: fatal error: rte_config.h: No such file or
directory
 #include <rte_config.h>
                        ^
compilation terminated.
make[2]: *** [vhost.o] Error 1
make[1]: *** [vhost] Error 2
make: *** [lib] Error 2

Thanks,
Gyan

On Sun, May 13, 2018 at 1:45 AM Shahar Salzman <shahar.salzman(a)kaminario.=
com>
wrote:

> Hi Daniel,
>
>
> I ran the dpdk build with T=3Dspdk-linuxapp-gcc and build works fine.
>
> Isn't config/common_spdk only for configuration options (no reference for
> this type of definition) ? I would have thought that the compilation flag
> should go in spdk/dpdkbuild/Makefile, or in one of the mk files?
>
>
> Shahar
>
>
>
> ------------------------------
> *From:* SPDK <spdk-bounces(a)lists.01.org> on behalf of Verkamp, Daniel <
> daniel.verkamp(a)intel.com>
> *Sent:* Thursday, May 10, 2018 8:31:21 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] Building spdk on CentOS6
>
>
> Hi Shahar,
>
>
>
> All of the config options you listed below are already disabled in the
> config that we use to build the DPDK subodule (see config/common_spdk in
> our DPDK branch).  Can you confirm that this works for you?
>
>
>
> I think we can add -fno-strict-aliasing to the DPDK config/common_spdk
> file as well if this is necessary to make it work on RHEL 6 (I am surpris=
ed
> DPDK doesn=E2=80=99t already compile with this enabled).  If you could up=
load a
> second patch for this, that would be great.
>
>
>
> Thanks,
>
> -- Daniel
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Shahar
> Salzman
> *Sent:* Thursday, May 10, 2018 6:39 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Building spdk on CentOS6
>
>
>
> Hi Daniel,
>
>
>
> I updated the patch as you suggested.
>
> How do you want to approach the gcc flag and the modules that spdk does
> not need?
>
> If you are building dpdk from an external makefile (as we are doing), all
> of the above can be added to the makefile with CONFIG...=3Dn and
> EXTRA_CFLAGS=3D"-fno-strict-aliasing", or I can send you another dpdk pat=
ch
> fixing this within dpdk, so there is no change in the builder, and maybe
> there is another option.
>
>
>
> WDYT?
>
>
>
> Shahar
>
>
> ------------------------------
>
> *From:* SPDK <spdk-bounces(a)lists.01.org> on behalf of Luse, Paul E <
> paul.e.luse(a)intel.com>
> *Sent:* Wednesday, May 9, 2018 8:19:33 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] Building spdk on CentOS6
>
>
>
> That=E2=80=99s great, thanks Daniel!
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01=
.org>]
> *On Behalf Of *Verkamp, Daniel
> *Sent:* Wednesday, May 9, 2018 9:43 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Building spdk on CentOS6
>
>
>
> I agree with Paul=E2=80=99s review; it looks like it shouldn=E2=80=99t be=
 a big problem to
> carry a patch for this in our DPDK submodule, at least while CentOS 6 is
> still officially supported. (I posted a few review comments on the patch,
> but the general approach looks fine.)
>
>
>
> The Intel build pool had a CentOS 6 test machine in the pool until
> recently, and we should be able to resurrect it to provide test coverage
> again.
>
>
>
> Thanks,
>
> -- Daniel
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01=
.org>]
> *On Behalf Of *Shahar Salzman
> *Sent:* Wednesday, May 9, 2018 6:12 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Building spdk on CentOS6
>
>
>
> Hi Paul,
>
>
>
> Thanks for the review.
>
> We are now working on CentOS7 support, but it is not complete yet, so we
> will be on CentOS6 about 6-12 months.
>
>
>
> Shahar
> ------------------------------
>
> *From:* SPDK <spdk-bounces(a)lists.01.org> on behalf of Luse, Paul E <
> paul.e.luse(a)intel.com>
> *Sent:* Wednesday, May 9, 2018 3:51:23 PM
> *To:* Storage Performance Development Kit
> *Subject:* Re: [SPDK] Building spdk on CentOS6
>
>
>
> Thanks for doing that! I put my 2 cents on the review, one more data point
> for the maintainers might help =E2=80=93 help long do you believe your te=
am will be
> on CentOS6? Don=E2=80=99t need a specific date but are we talking months,=
 years,
> indefinitely?
>
>
>
> I think the other implication of accepting this in the community would be
> to spin up a CentOS6 VM for some basic testing for as long as we hold that
> patch in our DPDK fork which is not a huge thing but want to make sure
> that=E2=80=99s clear as well. If nothing else we=E2=80=99ll know if somet=
hing else breaks
> it sooner than later and will need to decide, again, at that time if we
> pull it or if it=E2=80=99s a high enough priority for anyone to dig in an=
d find a
> potential solution like you did here.  Assuming, of course, that this one
> is accepted J
>
>
>
> Thanks,
>
> Paul
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01=
.org>]
> *On Behalf Of *Shahar Salzman
> *Sent:* Wednesday, May 9, 2018 2:01 AM
> *To:* spdk(a)lists.01.org
> *Subject:* Re: [SPDK] Building spdk on CentOS6
>
>
>
> Hi Ben,
>
>
>
> Pushed the fix to gerrithub -
> https://review.gerrithub.io/#/c/spdk/dpdk/+/410559/
>
>
>
> We are currently using a "frozen" dpdk tree based version 16.07, but I am
> pushing to using the spdk fork and performing regular updates so if things
> break, I will attempt to fix them, and post on the list.
>
>
>
> The above fixes a blunt gcc API issue, but there are more subtle inner
> module issues which I have not investigated, simply removed the modules a=
ll
> together as they are not needed by spdk.
>
>
>
> I can set up a howto for users who may need CentOS6 support or we can
> modify the config (as these modules are not really needed for spdk). For
> the sake of clarity here are the modules "configed out" (as I mentioned
> they are not used by spdk):
>
> +CONFIG_RTE_LIBRTE_PMD_BOND=3Dn
>
> +CONFIG_RTE_LIBRTE_CRYPTODEV=3Dn
>
> +CONFIG_RTE_LIBRTE_SECURITY=3Dn
>
> +CONFIG_RTE_LIBRTE_PMD_SW_EVENTDEV=3Dn
>
> +CONFIG_RTE_LIBRTE_PMD_OCTEONTX_SSOVF=3Dn
>
> +CONFIG_RTE_LIBRTE_LPM=3Dn
>
> +CONFIG_RTE_LIBRTE_FLOW_CLASSIFY=3Dn
>
> +CONFIG_RTE_LIBRTE_TABLE=3Dn
>
> +CONFIG_RTE_LIBRTE_PIPELINE=3Dn
>
> +CONFIG_RTE_TEST_PMD=3Dn
>
> +CONFIG_RTE_LIBRTE_PMD_TAP=3Dn
>
> +CONFIG_RTE_LIBRTE_AVP_PMD=3Dn
>
>
>
> Regarding the -fn-strict-aliasing issue, I'll see if I can do the same
> trick I did for the gcc alias in the Makefile, and push the fix for revie=
w.
>
>
>
> Shahar
>
>
> ------------------------------
>
> *From:* SPDK <spdk-bounces(a)lists.01.org> on behalf of Walker, Benjamin <
> benjamin.walker(a)intel.com>
> *Sent:* Tuesday, May 8, 2018 7:49:44 PM
> *To:* spdk(a)lists.01.org
> *Subject:* Re: [SPDK] Building spdk on CentOS6
>
>
>
> On Tue, 2018-05-08 at 14:34 +0000, Shahar Salzman wrote:
> > Got it.
> >
> > I was tracking the dpdk project, and focused on the branch that Daniel
> told me
> > to work on.
> > So would the correct process be to push the patch to the spdk/dpdk
> gerrithub
> > and discuss the commit, or the other way around?
>
> Yes - go ahead and push to spdk/dpdk on GerritHub so we can all review it.
> The
> SPDK project has never written out a policy on which versions of DPDK are
> supported for any given SPDK release. The code is always tested using the
> latest
> released DPDK, and then attempts to support older versions of DPDK are
> done in
> an ad hoc manner. Similarly, the project never explicitly states which
> operating
> systems are supported. Instead, we indicate which operating systems the
> code is
> regularly tested on (Fedora N and N-1, Ubuntu latest, Ubuntu LTS, CentOS =
7,
> FreeBSD 11).
>
> I see that CentOS 6 is no longer receiving full updates and is in
> maintenance
> only mode. It will stop receiving even maintenance updates in 2020. I
> think that
> is indicative of the level of support SPDK should provide here.
>
> Given the above, if this patch can be applied to the DPDK fork and doesn't
> have
> any additional impact, I'm not opposed to applying it. That wouldn't, in =
my
> mind, mean that SPDK is committed to supporting CentOS 6 indefinitely. But
> certainly that support could continue while it is relatively easy and
> convenient.
>
> >
> > From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Luse, Paul E <
> paul.e.luse@
> <paul.e.luse@%0b>> intel.com>
> > Sent: Tuesday, May 8, 2018 5:14:10 PM
> > To: Storage Performance Development Kit
> > Subject: Re: [SPDK] Building spdk on CentOS6
> >
> > Sure, I understand that.  SPDK maintains a fork of dpdk in github at
> https://g
> > ithub.com/spdk/dpdk for the purpose of temporarily putting things in
> that we
> > need that can=E2=80=99t make it upstream yet for whatever reason =E2=80=
=93 this is the
> > submodule in the SPDK repo.  I=E2=80=99m not saying that it makes sense=
 for your
> patch
> > to land there permanently but it does make sense to talk about it.
> >
> > No problem on not being on the call, we=E2=80=99ll get some input from =
the
> maintainers
> > and others I=E2=80=99m sure=E2=80=A6
> >
> > Thx
> > Paul
> >
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01=
.org>]
> On Behalf Of Shahar Salzman
> > Sent: Tuesday, May 8, 2018 7:04 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Building spdk on CentOS6
> >
> > Hey Paul,
> >
> > The problem is that the patch is in dpdk and not in spdk, and this is t=
he
> > reason I offered to support it.
> > The patch (bellow) is only a few lines long modifying the deprecated
> attribute
> > definition, a modification to the config, and an extra compilation flag.
> > All of the above are required in order to build dpdk 18.01 on CentOS6,
> there
> > are no problems building spdk on CentOS6.
> >
> > I understood that you are already applying minor patches to dpdk, so
> question
> > is, whether this patch can be added?
> >
> > Unfortunately due to some personal issues, I am unable to participate in
> > today's meeting.
> >
> > Thanks,
> > Shahar
> > From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Luse, Paul E <
> paul.e.luse@
> <paul.e.luse@%0b>> intel.com>
> > Sent: Tuesday, May 8, 2018 4:25:29 PM
> > To: Storage Performance Development Kit
> > Subject: Re: [SPDK] Building spdk on CentOS6
> >
> > Hi Shahar,
> >
> > Couple of quick things:
> >
> > * thanks for putting all this time and effort into trying to make this
> work!
> > * in the community model, patches aren=E2=80=99t accepted based on one
> individual=E2=80=99s
> > ability/desire to support them so appreciate the offer but in general a=
ny
> > patch that is accepted becomes the responsibly of the community (mostly
> the
> > maintainers) to support it long term.  Otherwise we=E2=80=99d have what=
 I like
> to call
> > =E2=80=9Cthe flea market model=E2=80=9D where there=E2=80=99s a bunch o=
f separate individuals
> > supporting their own things with very little cohesiveness across
> everything J
> >
> > So, wrt next steps, if the patch isn=E2=80=99t a tremendous amount of e=
ffort I
> would
> > suggest you go ahead and submit it. That=E2=80=99s the best way to get =
everyone
> on the
> > same page wrt exactly what we=E2=80=99re talking about.  If it is some
> significant
> > effort then, as Pawel states, you can call into a community meeting (see
> email
> > I sent out last night) or you can explain more details on this list.
> >
> > Thanks again!!
> > Paul
> >
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01=
.org>]
> On Behalf Of Wodkowski, PawelX
> > Sent: Tuesday, May 8, 2018 5:57 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Building spdk on CentOS6
> >
> > I think someone will answer you on this mailing list but there is
> community
> > meeting today too. You can raise this if you wish.
> >
> > Pawel
> >
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01=
.org>]
> On Behalf Of Shahar Salzman
> > Sent: Tuesday, May 8, 2018 2:23 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Building spdk on CentOS6
> >
> > Hi,
> >
> > I attempted to push the gcc support patch to dpdk, and got rejected
> since they
> > do not wish to support CentOS6.
> > Would it be possible to manually add the patch to the supported dpdk
> version
> > (currently 18.01).
> > I would be happy to support CentOS6 issues.
> >
> > Thanks,
> > Shahar
> >
> > From: Shahar Salzman
> > Sent: Tuesday, April 24, 2018 3:40:51 PM
> > To: Storage Performance Development Kit
> > Cc: Ilan Steinberg; Ido Benda; Yael Shavit
> > Subject: Re: Building spdk on CentOS6
> >
> > OK, got everything to work.
> >
> > For DPDK, I had to to do the following:
> > add a patch to make the new __rte_experimental attribute to work
> > remove a bunch of modules via config which had some gnarly compilation
> errors,
> > and are not needed for spdk
> > use -fno-strict-aliasing in the EXTRA_CFLAGS to avoid a ton of strict
> aliasing
> > errors.
> >
> > I tried spdk 18.01, and ran into some issues with the shared library
> build. As
> > I saw that Daniel had already dealt with them, I tried cherry picking,
> but
> > eventually used the master branch.
> > I got these type of warnings:
> >
> > /usr/bin/ld: warning: librte_mbuf.so.3.1, needed by
> > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp-
> > gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link)
> > /usr/bin/ld: warning: librte_ethdev.so.8.1, needed by
> > /khome/shahar.salzman/Kaminario/git/dpdk/x86_64-native-linuxapp-
> > gcc/lib/librte_bus_pci.so, not found (try using -rpath or -rpath-link)
> >
> > I added the following patch bellow to make them go away.
> >
> > I'll approach the dpdk community on the mailing list in order to get the
> > __rte_experimental stuff to work.
> > My final configuration is:
> > CenrOS6.4
> > gcc 4.4.7-17
> > dpdk v18.02
> > spdk master
> >
> > What are the next steps?
> >
> > Shahar
> >
> > P.S. Here is the patch to remove the warnings above.
> > commit d4ef744de1cc4928972b7042d1c90aff12db7867
> > Author: shahar salzman <shahar.salzman(a)kaminario.com>
> > Date:   Tue Apr 24 14:31:52 2018 +0300
> >
> >     lib/env_dpdk: add required libraries
> >
> > diff --git a/lib/env_dpdk/env.mk b/lib/env_dpdk/env.mk
> > index 450043c..b46bfed 100644
> > --- a/lib/env_dpdk/env.mk
> > +++ b/lib/env_dpdk/env.mk
> > @@ -68,6 +68,18 @@ ifneq ($(wildcard
> $(DPDK_ABS_DIR)/lib/librte_malloc.*),)
> >  DPDK_LIB_LIST +=3D rte_malloc
> >  endif
> >
> > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_mbuf.*))
> > +DPDK_LIB_LIST +=3D rte_mbuf
> > +endif
> > +
> > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_net.*))
> > +DPDK_LIB_LIST +=3D rte_net
> > +endif
> > +
> > +ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_ethdev.*))
> > +DPDK_LIB_LIST +=3D rte_ethdev
> > +endif
> > +
> >  # librte_pci and librte_bus_pci were added in DPDK 17.11. Link these
> > libraries conditionally
> >  # based on their existence to maintain backward compatibility.
> >  ifneq (, $(wildcard $(DPDK_ABS_DIR)/lib/librte_pci.*))
> >
> >
> >
> > From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Shahar Salzman
> <shahar.sal
> > zman(a)kaminario.com>
> > Sent: Tuesday, April 24, 2018 9:57 AM
> > To: Storage Performance Development Kit
> > Subject: Re: [SPDK] Building spdk on CentOS6
> >
> > Great, and thanks!
> >
> > I have been running into issues compiling dpdk 17.11 on CentOS6 (there
> seems
> > to be a gcc issue that does not exist in other versions), I'll focus my
> > efforts on dpdk 18.02.
> > From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Verkamp, Daniel <
> daniel.ve
> > rkamp(a)intel.com>
> > Sent: Monday, April 23, 2018 9:40:36 PM
> > To: Storage Performance Development Kit
> > Subject: Re: [SPDK] Building spdk on CentOS6
> >
> > Hi Shahar,
> >
> > I=E2=80=99ve pushed a patch for the SPDK master branch that should allo=
w it to
> compile
> > with DPDK 16.07: https://review.gerrithub.io/#/c/spdk/spdk/+/408743/
> >
> > Gerrit Code Review
> >
> > review.gerrithub.io
> >
> > Keep in touch. Copyright =C2=A9 2017 | GerritForge Ltd. info(a)gerritfo=
rge.com
> > www.gerritforge.com
> >
> >
> > However, DPDK 16.07 is no longer supported upstream; I highly recommend
> that
> > you update to a newer version. SPDK is currently tested with DPDK 18.02.
> >
> > The VFIO issue you mention has also been fixed on SPDK master, and it
> will be
> > part of the SPDK v18.04 release.
> >
> > Thanks,
> > -- Daniel
> >
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01=
.org>]
> On Behalf Of Shahar Salzman
> > Sent: Sunday, April 22, 2018 6:41 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Building spdk on CentOS6
> >
> > Hi,
> >
> > Finally got to looking at support of spdk build on CentOS6, things look
> good,
> > except for one issue.
> > spdk is latest 18.01.x version, dpdk is 16.07 (+3 dpdk patches to allow
> > compilation) and some minor patches (mainly some memory configuration
> stuff),
> > kernel is a patched 4.9.6.
> >
> > build succeeded except for the usage of the dpdk function
> pci_vfio_is_enabled.
> > I had to apply the patch bellow, removing the usage of this function and
> then
> > compilation completed without any issues.
> >
> > It seems that I am missing some sort of dpdk configuration as I see that
> the
> > function is built, but not packaged into the generated archive.
> >
> > I went back to square one and ran the instructions in
> http://www.spdk.io/doc/g
> > etting_started.html, but I see no mention of dpdk there. Actually the
> > ./configure requires it.
> >
> > My next step is to use a more recent dpdk, but shouldn't this work with
> my
> > version? Am I missing some dpdk configuration?
> > BTW, as we are not using vhost, on our 17.07 version we simply use
> > CONFIG_VHOST=3Dn in order to skip this, but I would be happier if we us=
ed a
> > better solution.
> >
> > Shahar
> >
> > P.S. Here is the patch to remove use of this function:
> > diff --git a/lib/env_dpdk/vtophys.c b/lib/env_dpdk/vtophys.c
> > index 92aa256..f38929f 100644
> > --- a/lib/env_dpdk/vtophys.c
> > +++ b/lib/env_dpdk/vtophys.c
> > @@ -53,8 +53,10 @@
> >  #define SPDK_VFIO_ENABLED 1
> >  #include <linux/vfio.h>
> >
> > +#if 0
> >  /* Internal DPDK function forward declaration */
> >  int pci_vfio_is_enabled(void);
> > +#endif
> >
> >  struct spdk_vfio_dma_map {
> >         struct vfio_iommu_type1_dma_map map;
> > @@ -341,9 +343,11 @@ spdk_vtophys_iommu_init(void)
> >         DIR *dir;
> >         struct dirent *d;
> >
> > +#if 0
> >         if (!pci_vfio_is_enabled()) {
> >                 return;
> >         }
> > +#endif
> >
> >         dir =3D opendir("/proc/self/fd");
> >         if (!dir) {
> >
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

--===============4314877694713167070==--