DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
From: Bruce Richardson <bruce.richardson@intel.com>
To: Andrzej Ostruszka <amo@semihalf.com>
Cc: Neil Horman <nhorman@tuxdriver.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	Ray Kinsella <mdr@ashroe.eu>,
	dev@dpdk.org, Aaron Conole <aconole@redhat.com>,
	Michael Santana <msantana@redhat.com>,
	John McNamara <john.mcnamara@intel.com>,
	Marko Kovacevic <marko.kovacevic@intel.com>,
	David Hunt <david.hunt@intel.com>,
	Vladimir Medvedkin <vladimir.medvedkin@intel.com>,
	Robert Sanford <rsanford@akamai.com>,
	Erik Gabriel Carrillo <erik.g.carrillo@intel.com>,
	mattias.ronnblom@ericsson.com, stephen@networkplumber.org,
	Andrzej Ostruszka <aostruszka@marvell.com>,
	bluca@debian.org
Subject: Re: [dpdk-dev] [PATCH v2 01/10] build: add an option to enable LTO build
Date: Fri, 27 Sep 2019 20:55:31 +0100
Message-ID: <20190927195531.GA1871@bricha3-MOBL.ger.corp.intel.com> (raw)
In-Reply-To: <a4b8f209-a8ed-57a0-0038-4e073a4216b8@semihalf.com>

On Thu, Sep 26, 2019 at 05:32:12PM +0200, Andrzej Ostruszka wrote:
> On 9/24/19 2:59 PM, Neil Horman wrote:
> > On Tue, Sep 24, 2019 at 12:25:35PM +0200, Bruce Richardson wrote:
> >> On Tue, Sep 24, 2019 at 08:46:25AM +0200, Andrzej Ostruszka wrote:
> >>> On 9/23/19 6:13 PM, Bruce Richardson wrote:
> [...]
> >> The real issue seems to be that the compat.h header has different
> >> compilation paths for static and shared libraries, which means that any C
> >> file including it can't have a .o file that can be used for both a .a and a
> >> .so simultaneously. Having it default to the shared library path seems to
> >> work fine thus far, but with LTO it seems broken as you say. Adding Neil as
> >> the original file author of this in case he has any suggestions here. I'd
> >> really rather not have to go back to building .a's and .so's separately.
> >>
> > The notion of using the same object file to link to a static archive and a dso
> > seems somewhat suspect to me in general.
> 
> I'd think so too ... but there might be something fishy with gcc here.
> More on this below.
> 
> [...]
> > That said, if the goal is to just overcome this particular situation, it might
> > (strong might), be sufficient to simply augment the MAP_STATIC_SYMBOL macro in
> > the CONFIG_RTE_BUILD_SHARED_LIB=n case to append the 'used' attribute.
> > Ostensibly, LTO would be smart enough then to not eliminate the symbol?  Just a
> > thought.
> Just to clarify things here is the current status in a nutshell.
> 1. The make based build work with LTO (both static and shared).  It is
> using CONFIG_*_SHARED_LIB option and thus different paths of
> rte_compat.h are used.
> 
> 2. The meson build is not using config and has "SHARED" fixed to "y" in
> config/rte_config.h.  This works and produces:
> - shared library that is working when linking both w/ and w/o LTO
> - static library that is only working when linking w/o LTO - when
> linking with LTO then it complains about missing symbols for which
> different symbol versions are defined.
> 
> Augmenting MAP_STATIC_SYMBOL won't help since it is not used at all in
> meson build.
> 
> I've played around with couple ideas.  Some of them might sound stupid
> for you - but I'll report them anyway
> 
> Modifying symbol tables
> -----------------------
> I thought that since problems are only with versioned symbols then I'll
> try to modify the symbols tables archives so they look like in "static
> make" case e.g. (for just one symbol):
> 
> $ objcopy --strip-symbol=rte_timer_subsystem_init@DPDK_2.0 \
> 	--redefine-sym \
> rte_timer_subsystem_init@@DPDK_19.05=rte_timer_subsystem_init \
> 	librte_timer.a
> 
> After this the symbol table looks like in fully static case but this
> doesn't work so I'm pretty sure that when using LTO linker does not
> check the symbol table at all and just looks in "lto" sections.
> 
> Adding static macro
> -------------------
> I've added for the shared case macro:
> #define MAP_STATIC_SYMBOL(f, p) f \
> 	__attribute__((alias(RTE_STR(p)),weak))
> with the idea that maybe during linking against *.a versioned symbols
> are not taken into account and if I add weak non-versioned symbol then
> it will be used when linking against *.a and when linking against *.so
> the strong versioned one will be used.  This thinking is in contrast
> with the fact that meson build works w/o LTO where only versioned
> symbols are present in those problematic libs :) - and it doesn't work.
> Linker reports multiple definitions.
> 
> So adding these two together it seems to me that:
> a) gcc is using only internal "lto" sections when linking with LTO
> b) gcc is storing those versioned symbols in those "lto" sections
> c) when linking w/ LTO against archives is not capable to use those
> versioned symbols in "lto" sections
> d) when linking w/ LTO against shared libraries is capable to use
> versioned symbols from "lto" sections
> e) when linking w/o LTO against archive is capable to use versioned
> symbols from global symbol table (in contrast with point 'c' above)
> 
> I'd appreciate some input from those who know gcc internals.  In the
> meantime I'll try to come up with minimal example and follow up on gcc
> related lists/groups.
> 
Please have a look over the patchset I just posted, as one possible
solution to this. http://patches.dpdk.org/project/dpdk/list/?series=6594

For the general case of compiling DPDK, a given block of C code is going to
result in the same object file whether compiled for static or shared (so
long as -fPIC is passed to the compile), so running two compiles for each
and every C file would be wasted effort. However, for those files with
function versioning, I think any proper solution has to involve compiling
twice with different macros, so that's what the patchset seeks to achieve.

/Bruce

  reply index

Thread overview: 110+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-17  7:57 [dpdk-dev] [PATCH v2 00/10] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 01/10] build: add an option to enable LTO build Andrzej Ostruszka
2019-09-18 10:36   ` Bruce Richardson
2019-09-18 13:32   ` Ray Kinsella
2019-09-19 12:35     ` Andrzej Ostruszka
2019-09-19 13:28       ` Ray Kinsella
2019-09-19 15:16         ` Bruce Richardson
2019-09-20  7:38           ` Ray Kinsella
2019-09-23  7:23             ` Thomas Monjalon
2019-09-23  9:36               ` Ray Kinsella
2019-09-23 10:16                 ` Mattias Rönnblom
2019-09-23 12:03               ` Andrzej Ostruszka
2019-09-23 12:06                 ` Bruce Richardson
2019-09-23 13:02                   ` Andrzej Ostruszka
2019-09-23 16:13                     ` Bruce Richardson
2019-09-24  6:46                       ` Andrzej Ostruszka
2019-09-24 10:25                         ` Bruce Richardson
2019-09-24 11:52                           ` Andrzej Ostruszka
2019-09-24 12:11                             ` Bruce Richardson
2019-09-24 12:59                           ` Neil Horman
2019-09-24 16:01                             ` Ray Kinsella
2019-09-26 15:32                             ` Andrzej Ostruszka
2019-09-27 19:55                               ` Bruce Richardson [this message]
2019-09-23 12:16                 ` Ray Kinsella
2019-10-27 11:31     ` Thomas Monjalon
2019-10-28  8:36       ` Andrzej Ostruszka
2019-10-28  9:07         ` Thomas Monjalon
2019-10-28 12:12         ` Andrzej Ostruszka
2019-10-28 17:16           ` Thomas Monjalon
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 02/10] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-12 13:35   ` Jerin Jacob
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 03/10] app/eventdev: fix maybe-uninitialized warnings for LTO build Andrzej Ostruszka
2019-10-12 13:52   ` Jerin Jacob
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 04/10] event/octeontx2: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 05/10] app/test: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 06/10] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 07/10] net/e1000: fix maybe-uninitialized warnings for LTO build Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 08/10] net/i40e: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 09/10] net/ifc: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 10/10] net/qede: " Andrzej Ostruszka
     [not found] ` <20191021105707.25691-1-aostruszka@marvell.com>
     [not found]   ` <20191021105707.25691-2-aostruszka@marvell.com>
2019-10-21 12:59     ` [dpdk-dev] [PATCH v3 01/10] build: add an option to enable " Bruce Richardson
2019-10-22  8:53       ` Andrzej Ostruszka
2019-10-22 11:54 ` [dpdk-dev] [PATCH v4 00/10] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 01/10] build: add an option to enable LTO build Andrzej Ostruszka
2019-10-22 12:45     ` Bruce Richardson
2019-10-27 11:47     ` Thomas Monjalon
2019-10-28 10:47       ` Andrzej Ostruszka
2019-10-28 11:03         ` Thomas Monjalon
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 02/10] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 03/10] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 04/10] event/octeontx2: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 05/10] app/test: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 06/10] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 07/10] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 08/10] net/i40e: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 09/10] net/ifc: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 10/10] net/qede: " Andrzej Ostruszka
2019-10-22 12:48   ` [dpdk-dev] [PATCH v4 00/10] Add an option to use LTO for DPDK build Bruce Richardson
2019-10-22 13:03     ` Andrzej Ostruszka
2019-10-28 14:21   ` [dpdk-dev] [PATCH v5 00/11] " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 01/11] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-10-29 10:49       ` Neil Horman
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 02/11] build: add an option to enable LTO build Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 03/11] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 04/11] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 05/11] event/octeontx2: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 06/11] app/test: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 07/11] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 08/11] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 09/11] net/i40e: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 10/11] net/ifc: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 11/11] net/qede: " Andrzej Ostruszka
2019-10-29 14:12     ` [dpdk-dev] [PATCH v6 00/12] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 01/12] doc: fix description of versioning macros Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 02/12] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 03/12] build: add an option to enable LTO build Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 04/12] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 05/12] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 06/12] event/octeontx2: " Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 07/12] app/test: " Andrzej Ostruszka
2019-11-01 17:15         ` Wang, Yipeng1
2019-11-04 13:48           ` Andrzej Ostruszka
2019-11-07 17:48             ` Wang, Yipeng1
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 08/12] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-11-04 11:46         ` Hemant Agrawal
2019-11-04 14:33           ` Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 09/12] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 10/12] net/i40e: " Andrzej Ostruszka
2019-11-01  2:05         ` Xing, Beilei
2019-11-04 14:06           ` Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 11/12] net/ifc: " Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 12/12] net/qede: " Andrzej Ostruszka
2019-10-30  9:09       ` [dpdk-dev] [PATCH v6 00/12] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-30 14:23         ` Aaron Conole
2019-11-07 15:03       ` [dpdk-dev] [PATCH v7 " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 01/12] doc: fix description of versioning macros Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 02/12] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 03/12] build: add an option to enable LTO build Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 04/12] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 05/12] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 06/12] event/octeontx2: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 07/12] app/test: " Andrzej Ostruszka
2019-11-07 17:53           ` Wang, Yipeng1
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 08/12] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 09/12] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 10/12] net/i40e: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 11/12] net/ifc: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 12/12] net/qede: " Andrzej Ostruszka
2019-11-08 14:24         ` [dpdk-dev] [PATCH v7 00/12] Add an option to use LTO for DPDK build Thomas Monjalon
2019-11-01 21:33 ` [dpdk-dev] [PATCH v2 00/10] " Stephen Hemminger

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190927195531.GA1871@bricha3-MOBL.ger.corp.intel.com \
    --to=bruce.richardson@intel.com \
    --cc=aconole@redhat.com \
    --cc=amo@semihalf.com \
    --cc=aostruszka@marvell.com \
    --cc=bluca@debian.org \
    --cc=david.hunt@intel.com \
    --cc=dev@dpdk.org \
    --cc=erik.g.carrillo@intel.com \
    --cc=john.mcnamara@intel.com \
    --cc=marko.kovacevic@intel.com \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=mdr@ashroe.eu \
    --cc=msantana@redhat.com \
    --cc=nhorman@tuxdriver.com \
    --cc=rsanford@akamai.com \
    --cc=stephen@networkplumber.org \
    --cc=thomas@monjalon.net \
    --cc=vladimir.medvedkin@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
		dev@dpdk.org
	public-inbox-index dpdk-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git