DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
From: Andrzej Ostruszka <amo@semihalf.com>
To: Neil Horman <nhorman@tuxdriver.com>,
	Bruce Richardson <bruce.richardson@intel.com>
Cc: Thomas Monjalon <thomas@monjalon.net>,
	Ray Kinsella <mdr@ashroe.eu>,
	dev@dpdk.org, Aaron Conole <aconole@redhat.com>,
	Michael Santana <msantana@redhat.com>,
	John McNamara <john.mcnamara@intel.com>,
	Marko Kovacevic <marko.kovacevic@intel.com>,
	David Hunt <david.hunt@intel.com>,
	Vladimir Medvedkin <vladimir.medvedkin@intel.com>,
	Robert Sanford <rsanford@akamai.com>,
	Erik Gabriel Carrillo <erik.g.carrillo@intel.com>,
	mattias.ronnblom@ericsson.com, stephen@networkplumber.org,
	Andrzej Ostruszka <aostruszka@marvell.com>
Subject: Re: [dpdk-dev] [PATCH v2 01/10] build: add an option to enable LTO build
Date: Thu, 26 Sep 2019 17:32:12 +0200
Message-ID: <a4b8f209-a8ed-57a0-0038-4e073a4216b8@semihalf.com> (raw)
In-Reply-To: <20190924125942.GA6041@hmswarspite.think-freely.org>

On 9/24/19 2:59 PM, Neil Horman wrote:
> On Tue, Sep 24, 2019 at 12:25:35PM +0200, Bruce Richardson wrote:
>> On Tue, Sep 24, 2019 at 08:46:25AM +0200, Andrzej Ostruszka wrote:
>>> On 9/23/19 6:13 PM, Bruce Richardson wrote:
[...]
>> The real issue seems to be that the compat.h header has different
>> compilation paths for static and shared libraries, which means that any C
>> file including it can't have a .o file that can be used for both a .a and a
>> .so simultaneously. Having it default to the shared library path seems to
>> work fine thus far, but with LTO it seems broken as you say. Adding Neil as
>> the original file author of this in case he has any suggestions here. I'd
>> really rather not have to go back to building .a's and .so's separately.
>>
> The notion of using the same object file to link to a static archive and a dso
> seems somewhat suspect to me in general.

I'd think so too ... but there might be something fishy with gcc here.
More on this below.

[...]
> That said, if the goal is to just overcome this particular situation, it might
> (strong might), be sufficient to simply augment the MAP_STATIC_SYMBOL macro in
> the CONFIG_RTE_BUILD_SHARED_LIB=n case to append the 'used' attribute.
> Ostensibly, LTO would be smart enough then to not eliminate the symbol?  Just a
> thought.
Just to clarify things here is the current status in a nutshell.
1. The make based build work with LTO (both static and shared).  It is
using CONFIG_*_SHARED_LIB option and thus different paths of
rte_compat.h are used.

2. The meson build is not using config and has "SHARED" fixed to "y" in
config/rte_config.h.  This works and produces:
- shared library that is working when linking both w/ and w/o LTO
- static library that is only working when linking w/o LTO - when
linking with LTO then it complains about missing symbols for which
different symbol versions are defined.

Augmenting MAP_STATIC_SYMBOL won't help since it is not used at all in
meson build.

I've played around with couple ideas.  Some of them might sound stupid
for you - but I'll report them anyway

Modifying symbol tables
-----------------------
I thought that since problems are only with versioned symbols then I'll
try to modify the symbols tables archives so they look like in "static
make" case e.g. (for just one symbol):

$ objcopy --strip-symbol=rte_timer_subsystem_init@DPDK_2.0 \
	--redefine-sym \
rte_timer_subsystem_init@@DPDK_19.05=rte_timer_subsystem_init \
	librte_timer.a

After this the symbol table looks like in fully static case but this
doesn't work so I'm pretty sure that when using LTO linker does not
check the symbol table at all and just looks in "lto" sections.

Adding static macro
-------------------
I've added for the shared case macro:
#define MAP_STATIC_SYMBOL(f, p) f \
	__attribute__((alias(RTE_STR(p)),weak))
with the idea that maybe during linking against *.a versioned symbols
are not taken into account and if I add weak non-versioned symbol then
it will be used when linking against *.a and when linking against *.so
the strong versioned one will be used.  This thinking is in contrast
with the fact that meson build works w/o LTO where only versioned
symbols are present in those problematic libs :) - and it doesn't work.
Linker reports multiple definitions.

So adding these two together it seems to me that:
a) gcc is using only internal "lto" sections when linking with LTO
b) gcc is storing those versioned symbols in those "lto" sections
c) when linking w/ LTO against archives is not capable to use those
versioned symbols in "lto" sections
d) when linking w/ LTO against shared libraries is capable to use
versioned symbols from "lto" sections
e) when linking w/o LTO against archive is capable to use versioned
symbols from global symbol table (in contrast with point 'c' above)

I'd appreciate some input from those who know gcc internals.  In the
meantime I'll try to come up with minimal example and follow up on gcc
related lists/groups.

Regards
Andrzej

  parent reply index

Thread overview: 110+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-17  7:57 [dpdk-dev] [PATCH v2 00/10] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 01/10] build: add an option to enable LTO build Andrzej Ostruszka
2019-09-18 10:36   ` Bruce Richardson
2019-09-18 13:32   ` Ray Kinsella
2019-09-19 12:35     ` Andrzej Ostruszka
2019-09-19 13:28       ` Ray Kinsella
2019-09-19 15:16         ` Bruce Richardson
2019-09-20  7:38           ` Ray Kinsella
2019-09-23  7:23             ` Thomas Monjalon
2019-09-23  9:36               ` Ray Kinsella
2019-09-23 10:16                 ` Mattias Rönnblom
2019-09-23 12:03               ` Andrzej Ostruszka
2019-09-23 12:06                 ` Bruce Richardson
2019-09-23 13:02                   ` Andrzej Ostruszka
2019-09-23 16:13                     ` Bruce Richardson
2019-09-24  6:46                       ` Andrzej Ostruszka
2019-09-24 10:25                         ` Bruce Richardson
2019-09-24 11:52                           ` Andrzej Ostruszka
2019-09-24 12:11                             ` Bruce Richardson
2019-09-24 12:59                           ` Neil Horman
2019-09-24 16:01                             ` Ray Kinsella
2019-09-26 15:32                             ` Andrzej Ostruszka [this message]
2019-09-27 19:55                               ` Bruce Richardson
2019-09-23 12:16                 ` Ray Kinsella
2019-10-27 11:31     ` Thomas Monjalon
2019-10-28  8:36       ` Andrzej Ostruszka
2019-10-28  9:07         ` Thomas Monjalon
2019-10-28 12:12         ` Andrzej Ostruszka
2019-10-28 17:16           ` Thomas Monjalon
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 02/10] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-12 13:35   ` Jerin Jacob
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 03/10] app/eventdev: fix maybe-uninitialized warnings for LTO build Andrzej Ostruszka
2019-10-12 13:52   ` Jerin Jacob
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 04/10] event/octeontx2: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 05/10] app/test: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 06/10] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 07/10] net/e1000: fix maybe-uninitialized warnings for LTO build Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 08/10] net/i40e: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 09/10] net/ifc: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 10/10] net/qede: " Andrzej Ostruszka
     [not found] ` <20191021105707.25691-1-aostruszka@marvell.com>
     [not found]   ` <20191021105707.25691-2-aostruszka@marvell.com>
2019-10-21 12:59     ` [dpdk-dev] [PATCH v3 01/10] build: add an option to enable " Bruce Richardson
2019-10-22  8:53       ` Andrzej Ostruszka
2019-10-22 11:54 ` [dpdk-dev] [PATCH v4 00/10] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 01/10] build: add an option to enable LTO build Andrzej Ostruszka
2019-10-22 12:45     ` Bruce Richardson
2019-10-27 11:47     ` Thomas Monjalon
2019-10-28 10:47       ` Andrzej Ostruszka
2019-10-28 11:03         ` Thomas Monjalon
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 02/10] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 03/10] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 04/10] event/octeontx2: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 05/10] app/test: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 06/10] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 07/10] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 08/10] net/i40e: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 09/10] net/ifc: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 10/10] net/qede: " Andrzej Ostruszka
2019-10-22 12:48   ` [dpdk-dev] [PATCH v4 00/10] Add an option to use LTO for DPDK build Bruce Richardson
2019-10-22 13:03     ` Andrzej Ostruszka
2019-10-28 14:21   ` [dpdk-dev] [PATCH v5 00/11] " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 01/11] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-10-29 10:49       ` Neil Horman
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 02/11] build: add an option to enable LTO build Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 03/11] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 04/11] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 05/11] event/octeontx2: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 06/11] app/test: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 07/11] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 08/11] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 09/11] net/i40e: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 10/11] net/ifc: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 11/11] net/qede: " Andrzej Ostruszka
2019-10-29 14:12     ` [dpdk-dev] [PATCH v6 00/12] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 01/12] doc: fix description of versioning macros Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 02/12] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 03/12] build: add an option to enable LTO build Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 04/12] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 05/12] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 06/12] event/octeontx2: " Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 07/12] app/test: " Andrzej Ostruszka
2019-11-01 17:15         ` Wang, Yipeng1
2019-11-04 13:48           ` Andrzej Ostruszka
2019-11-07 17:48             ` Wang, Yipeng1
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 08/12] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-11-04 11:46         ` Hemant Agrawal
2019-11-04 14:33           ` Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 09/12] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 10/12] net/i40e: " Andrzej Ostruszka
2019-11-01  2:05         ` Xing, Beilei
2019-11-04 14:06           ` Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 11/12] net/ifc: " Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 12/12] net/qede: " Andrzej Ostruszka
2019-10-30  9:09       ` [dpdk-dev] [PATCH v6 00/12] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-30 14:23         ` Aaron Conole
2019-11-07 15:03       ` [dpdk-dev] [PATCH v7 " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 01/12] doc: fix description of versioning macros Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 02/12] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 03/12] build: add an option to enable LTO build Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 04/12] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 05/12] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 06/12] event/octeontx2: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 07/12] app/test: " Andrzej Ostruszka
2019-11-07 17:53           ` Wang, Yipeng1
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 08/12] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 09/12] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 10/12] net/i40e: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 11/12] net/ifc: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 12/12] net/qede: " Andrzej Ostruszka
2019-11-08 14:24         ` [dpdk-dev] [PATCH v7 00/12] Add an option to use LTO for DPDK build Thomas Monjalon
2019-11-01 21:33 ` [dpdk-dev] [PATCH v2 00/10] " Stephen Hemminger

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a4b8f209-a8ed-57a0-0038-4e073a4216b8@semihalf.com \
    --to=amo@semihalf.com \
    --cc=aconole@redhat.com \
    --cc=aostruszka@marvell.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.hunt@intel.com \
    --cc=dev@dpdk.org \
    --cc=erik.g.carrillo@intel.com \
    --cc=john.mcnamara@intel.com \
    --cc=marko.kovacevic@intel.com \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=mdr@ashroe.eu \
    --cc=msantana@redhat.com \
    --cc=nhorman@tuxdriver.com \
    --cc=rsanford@akamai.com \
    --cc=stephen@networkplumber.org \
    --cc=thomas@monjalon.net \
    --cc=vladimir.medvedkin@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
		dev@dpdk.org
	public-inbox-index dpdk-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git