DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
From: Andrzej Ostruszka <aostruszka@marvell.com>
To: <dev@dpdk.org>, Aaron Conole <aconole@redhat.com>,
	Michael Santana <maicolgabriel@hotmail.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	John McNamara <john.mcnamara@intel.com>,
	Marko Kovacevic <marko.kovacevic@intel.com>
Cc: <mattias.ronnblom@ericsson.com>, <stephen@networkplumber.org>
Subject: [dpdk-dev] [PATCH v6 03/12] build: add an option to enable LTO build
Date: Tue, 29 Oct 2019 15:12:03 +0100
Message-ID: <20191029141212.4907-4-aostruszka@marvell.com> (raw)
In-Reply-To: <20191029141212.4907-1-aostruszka@marvell.com>

This patch adds an option to enable link time optimization.  In addition
to LTO option itself (-flto) fat-lto-objects are being used.  This is
because during the build pmdinfogen scans the generated ELF objects to
find this_pmd_name* symbol in symbol table.  Without fat-lto-objects gcc
produces ELF only with extra symbols for internal use during linking.

Signed-off-by: Andrzej Ostruszka <aostruszka@marvell.com>
Acked-by: Bruce Richardson <bruce.richarson@intel.com>
---
 .travis.yml                              |  7 +++++
 config/common_base                       |  5 +++
 config/meson.build                       | 13 ++++++++
 doc/guides/prog_guide/index.rst          |  1 +
 doc/guides/prog_guide/lto.rst            | 40 ++++++++++++++++++++++++
 doc/guides/rel_notes/release_19_11.rst   |  9 ++++++
 mk/toolchain/gcc/rte.toolchain-compat.mk |  4 +++
 mk/toolchain/gcc/rte.vars.mk             | 12 +++++++
 mk/toolchain/icc/rte.vars.mk             |  8 +++++
 9 files changed, 99 insertions(+)
 create mode 100644 doc/guides/prog_guide/lto.rst

diff --git a/.travis.yml b/.travis.yml
index 3d6ef2959..3cd746dba 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -34,6 +34,7 @@ env:
   - DEF_LIB="static" OPTS="-Denable_kmods=false"
   - DEF_LIB="shared" OPTS="-Denable_kmods=false"
   - DEF_LIB="shared" RUN_TESTS=1
+  - DEF_LIB="shared" OPTS="-Db_lto=true"
 
 matrix:
   include:
@@ -105,6 +106,12 @@ matrix:
       apt:
         packages:
           - *extra_packages
+  - env: DEF_LIB="shared" OPTS="-Db_lto=true" EXTRA_PACKAGES=1
+    compiler: gcc
+    addons:
+      apt:
+        packages:
+          - *extra_packages
 
 
 script: ./.ci/${TRAVIS_OS_NAME}-build.sh
diff --git a/config/common_base b/config/common_base
index b2be3d96a..0d1207166 100644
--- a/config/common_base
+++ b/config/common_base
@@ -49,6 +49,11 @@ CONFIG_RTE_FORCE_INTRINSICS=n
 #
 CONFIG_RTE_ARCH_STRICT_ALIGN=n
 
+#
+# Enable link time optimization
+#
+CONFIG_RTE_ENABLE_LTO=n
+
 #
 # Compile to share library
 #
diff --git a/config/meson.build b/config/meson.build
index e1ebdad26..2b1cb92e7 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -225,3 +225,16 @@ add_project_arguments('-D_GNU_SOURCE', language: 'c')
 if is_freebsd
 	add_project_arguments('-D__BSD_VISIBLE', language: 'c')
 endif
+
+if get_option('b_lto')
+	if cc.has_argument('-ffat-lto-objects')
+		add_project_arguments('-ffat-lto-objects', language: 'c')
+	else
+		error('compiler does not support fat LTO objects - please turn LTO off')
+	endif
+	# workaround for gcc bug 81440
+	if cc.get_id() == 'gcc' and cc.version().version_compare('<8.0')
+		add_project_arguments('-Wno-lto-type-mismatch', language: 'c')
+		add_project_link_arguments('-Wno-lto-type-mismatch', language: 'c')
+	endif
+endif
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 692409af8..dc4851c57 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -65,5 +65,6 @@ Programmer's Guide
     ext_app_lib_make_help
     perf_opt_guidelines
     writing_efficient_code
+    lto
     profile_app
     glossary
diff --git a/doc/guides/prog_guide/lto.rst b/doc/guides/prog_guide/lto.rst
new file mode 100644
index 000000000..50aecc9e5
--- /dev/null
+++ b/doc/guides/prog_guide/lto.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Marvell International Ltd.
+
+Link Time Optimization
+======================
+
+The DPDK supports compilation with link time optimization turned on.
+This depends obviously on the ability of the compiler to do "whole
+program" optimization at link time and is available only for compilers
+that support that feature.
+To be more specific, compiler (in addition to performing LTO) have to
+support creation of ELF objects containing both normal code and internal
+representation (called fat-lto-objects in gcc and icc).
+This is required since during build some code is generated by parsing
+produced ELF objects (pmdinfogen).
+
+The amount of performance gain that one can get from LTO depends on the
+compiler and the code that is being compiled.
+However LTO is also useful for additional code analysis done by the
+compiler.
+In particular due to interprocedural analysis compiler can produce
+additional warnings about variables that might be used uninitialized.
+Some of these warnings might be "false positives" though and you might
+need to explicitly initialize variable in order to silence the compiler.
+
+Please note that turning LTO on causes considerable extension of
+build time.
+
+When using make based build, link time optimization can be enabled for
+the whole DPDK by setting:
+
+.. code-block:: console
+    CONFIG_ENABLE_LTO=y
+
+in config file.
+For the meson based build it can be enabled by setting meson built-in
+'b_lto' option:
+
+.. code-block:: console
+    meson build -Db_lto=true
diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index ae8e7b2f0..b11abe2fe 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -231,6 +231,15 @@ New Features
   * Added a console command to testpmd app, ``show port (port_id) ptypes`` which
     gives ability to print port supported ptypes in different protocol layers.
 
+* **Added build support for Link Time Optimization.**
+
+  LTO is an optimization technique used by the compiler to perform whole
+  program analysis and optimization at link time.  In order to do that
+  compilers store their internal representation of the source code that
+  the linker uses at the final stage of compilation process.
+
+  See :doc:`../prog_guide/lto` for more information:
+
 
 Removed Items
 -------------
diff --git a/mk/toolchain/gcc/rte.toolchain-compat.mk b/mk/toolchain/gcc/rte.toolchain-compat.mk
index ea40a11c0..ad4fad83c 100644
--- a/mk/toolchain/gcc/rte.toolchain-compat.mk
+++ b/mk/toolchain/gcc/rte.toolchain-compat.mk
@@ -88,6 +88,10 @@ else
 		MACHINE_CFLAGS := $(filter-out -march% -mtune% -msse%,$(MACHINE_CFLAGS))
 	endif
 
+	ifeq ($(shell test $(GCC_VERSION) -lt 45 && echo 1), 1)
+		CONFIG_RTE_ENABLE_LTO=n
+	endif
+
 	# Disable thunderx PMD for gcc < 4.7
 	ifeq ($(shell test $(GCC_VERSION) -lt 47 && echo 1), 1)
 		CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD=d
diff --git a/mk/toolchain/gcc/rte.vars.mk b/mk/toolchain/gcc/rte.vars.mk
index b852fcfd7..9fc704193 100644
--- a/mk/toolchain/gcc/rte.vars.mk
+++ b/mk/toolchain/gcc/rte.vars.mk
@@ -62,6 +62,18 @@ endif
 # process cpu flags
 include $(RTE_SDK)/mk/toolchain/$(RTE_TOOLCHAIN)/rte.toolchain-compat.mk
 
+ifeq ($(CONFIG_RTE_ENABLE_LTO),y)
+# 'fat-lto' is used since pmdinfogen needs to have 'this_pmd_nameX'
+# exported in symbol table and without this option only internal
+# representation is present.
+TOOLCHAIN_CFLAGS += -flto -ffat-lto-objects
+TOOLCHAIN_LDFLAGS += -flto
+# workaround for GCC bug 81440
+ifeq ($(shell test $(GCC_VERSION) -lt 80 && echo 1), 1)
+WERROR_FLAGS += -Wno-lto-type-mismatch
+endif
+endif
+
 # workaround GCC bug with warning "missing initializer" for "= {0}"
 ifeq ($(shell test $(GCC_VERSION) -lt 47 && echo 1), 1)
 WERROR_FLAGS += -Wno-missing-field-initializers
diff --git a/mk/toolchain/icc/rte.vars.mk b/mk/toolchain/icc/rte.vars.mk
index aa1422bf1..8aa87aa1e 100644
--- a/mk/toolchain/icc/rte.vars.mk
+++ b/mk/toolchain/icc/rte.vars.mk
@@ -54,5 +54,13 @@ endif
 # process cpu flags
 include $(RTE_SDK)/mk/toolchain/$(RTE_TOOLCHAIN)/rte.toolchain-compat.mk
 
+ifeq ($(CONFIG_RTE_ENABLE_LTO),y)
+# 'fat-lto' is used since pmdinfogen needs to have 'this_pmd_nameX'
+# exported in symbol table and without this option only internal
+# representation is present.
+TOOLCHAIN_CFLAGS += -flto -ffat-lto-objects
+TOOLCHAIN_LDFLAGS += -flto
+endif
+
 export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
 export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
-- 
2.17.1


  parent reply index

Thread overview: 110+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-17  7:57 [dpdk-dev] [PATCH v2 00/10] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 01/10] build: add an option to enable LTO build Andrzej Ostruszka
2019-09-18 10:36   ` Bruce Richardson
2019-09-18 13:32   ` Ray Kinsella
2019-09-19 12:35     ` Andrzej Ostruszka
2019-09-19 13:28       ` Ray Kinsella
2019-09-19 15:16         ` Bruce Richardson
2019-09-20  7:38           ` Ray Kinsella
2019-09-23  7:23             ` Thomas Monjalon
2019-09-23  9:36               ` Ray Kinsella
2019-09-23 10:16                 ` Mattias Rönnblom
2019-09-23 12:03               ` Andrzej Ostruszka
2019-09-23 12:06                 ` Bruce Richardson
2019-09-23 13:02                   ` Andrzej Ostruszka
2019-09-23 16:13                     ` Bruce Richardson
2019-09-24  6:46                       ` Andrzej Ostruszka
2019-09-24 10:25                         ` Bruce Richardson
2019-09-24 11:52                           ` Andrzej Ostruszka
2019-09-24 12:11                             ` Bruce Richardson
2019-09-24 12:59                           ` Neil Horman
2019-09-24 16:01                             ` Ray Kinsella
2019-09-26 15:32                             ` Andrzej Ostruszka
2019-09-27 19:55                               ` Bruce Richardson
2019-09-23 12:16                 ` Ray Kinsella
2019-10-27 11:31     ` Thomas Monjalon
2019-10-28  8:36       ` Andrzej Ostruszka
2019-10-28  9:07         ` Thomas Monjalon
2019-10-28 12:12         ` Andrzej Ostruszka
2019-10-28 17:16           ` Thomas Monjalon
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 02/10] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-12 13:35   ` Jerin Jacob
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 03/10] app/eventdev: fix maybe-uninitialized warnings for LTO build Andrzej Ostruszka
2019-10-12 13:52   ` Jerin Jacob
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 04/10] event/octeontx2: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 05/10] app/test: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 06/10] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 07/10] net/e1000: fix maybe-uninitialized warnings for LTO build Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 08/10] net/i40e: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 09/10] net/ifc: " Andrzej Ostruszka
2019-09-17  7:57 ` [dpdk-dev] [PATCH v2 10/10] net/qede: " Andrzej Ostruszka
     [not found] ` <20191021105707.25691-1-aostruszka@marvell.com>
     [not found]   ` <20191021105707.25691-2-aostruszka@marvell.com>
2019-10-21 12:59     ` [dpdk-dev] [PATCH v3 01/10] build: add an option to enable " Bruce Richardson
2019-10-22  8:53       ` Andrzej Ostruszka
2019-10-22 11:54 ` [dpdk-dev] [PATCH v4 00/10] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 01/10] build: add an option to enable LTO build Andrzej Ostruszka
2019-10-22 12:45     ` Bruce Richardson
2019-10-27 11:47     ` Thomas Monjalon
2019-10-28 10:47       ` Andrzej Ostruszka
2019-10-28 11:03         ` Thomas Monjalon
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 02/10] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 03/10] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 04/10] event/octeontx2: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 05/10] app/test: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 06/10] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 07/10] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 08/10] net/i40e: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 09/10] net/ifc: " Andrzej Ostruszka
2019-10-22 11:54   ` [dpdk-dev] [PATCH v4 10/10] net/qede: " Andrzej Ostruszka
2019-10-22 12:48   ` [dpdk-dev] [PATCH v4 00/10] Add an option to use LTO for DPDK build Bruce Richardson
2019-10-22 13:03     ` Andrzej Ostruszka
2019-10-28 14:21   ` [dpdk-dev] [PATCH v5 00/11] " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 01/11] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-10-29 10:49       ` Neil Horman
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 02/11] build: add an option to enable LTO build Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 03/11] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 04/11] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 05/11] event/octeontx2: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 06/11] app/test: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 07/11] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 08/11] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 09/11] net/i40e: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 10/11] net/ifc: " Andrzej Ostruszka
2019-10-28 14:21     ` [dpdk-dev] [PATCH v5 11/11] net/qede: " Andrzej Ostruszka
2019-10-29 14:12     ` [dpdk-dev] [PATCH v6 00/12] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 01/12] doc: fix description of versioning macros Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 02/12] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-10-29 14:12       ` Andrzej Ostruszka [this message]
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 04/12] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 05/12] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 06/12] event/octeontx2: " Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 07/12] app/test: " Andrzej Ostruszka
2019-11-01 17:15         ` Wang, Yipeng1
2019-11-04 13:48           ` Andrzej Ostruszka
2019-11-07 17:48             ` Wang, Yipeng1
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 08/12] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-11-04 11:46         ` Hemant Agrawal
2019-11-04 14:33           ` Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 09/12] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 10/12] net/i40e: " Andrzej Ostruszka
2019-11-01  2:05         ` Xing, Beilei
2019-11-04 14:06           ` Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 11/12] net/ifc: " Andrzej Ostruszka
2019-10-29 14:12       ` [dpdk-dev] [PATCH v6 12/12] net/qede: " Andrzej Ostruszka
2019-10-30  9:09       ` [dpdk-dev] [PATCH v6 00/12] Add an option to use LTO for DPDK build Andrzej Ostruszka
2019-10-30 14:23         ` Aaron Conole
2019-11-07 15:03       ` [dpdk-dev] [PATCH v7 " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 01/12] doc: fix description of versioning macros Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 02/12] build: annotate versioned symbols with __vsym macro Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 03/12] build: add an option to enable LTO build Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 04/12] eventdev: fix possible use of uninitialized var Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 05/12] app/eventdev: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 06/12] event/octeontx2: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 07/12] app/test: " Andrzej Ostruszka
2019-11-07 17:53           ` Wang, Yipeng1
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 08/12] net/dpaa2: fix possible use of uninitialized vars Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 09/12] net/e1000: clean LTO build warnings (maybe-uninitialized) Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 10/12] net/i40e: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 11/12] net/ifc: " Andrzej Ostruszka
2019-11-07 15:03         ` [dpdk-dev] [PATCH v7 12/12] net/qede: " Andrzej Ostruszka
2019-11-08 14:24         ` [dpdk-dev] [PATCH v7 00/12] Add an option to use LTO for DPDK build Thomas Monjalon
2019-11-01 21:33 ` [dpdk-dev] [PATCH v2 00/10] " Stephen Hemminger

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191029141212.4907-4-aostruszka@marvell.com \
    --to=aostruszka@marvell.com \
    --cc=aconole@redhat.com \
    --cc=dev@dpdk.org \
    --cc=john.mcnamara@intel.com \
    --cc=maicolgabriel@hotmail.com \
    --cc=marko.kovacevic@intel.com \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=stephen@networkplumber.org \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
		dev@dpdk.org
	public-inbox-index dpdk-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git