All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v8 00/16] support profile-optimized build for Python
@ 2017-11-27 12:06 Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 01/16] python3-native: support profile optimized build Markus Lehtonen
                   ` (15 more replies)
  0 siblings, 16 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

Changes since v7:
- simple selftest added


The following changes since commit 33418ed064fe9cff5b4803f09135a81d9170c189:

  runqemu: Also specialcase resolution of '.' to the file's location (2017-11-21 17:58:36 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-9338
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338

Markus Lehtonen (16):
  python3-native: support profile optimized build
  python3: fix depends of python3-tests
  python3: add python3-profile-opt recipe
  python3-profile-opt: rename libpython3
  openssh: extend to -native
  libmnl: enable native
  iptables: enable native
  iproute2: enable native
  oeqa/targetcontrol: re-introduce get_target_controller()
  oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__
  devtools/images: add python-pgo-image
  python3: support profile optimized build
  python3: fix profile-optimized build of modules
  python3: add python3-tools subpackage
  python3: remove two setup.py cross-compile hacks
  oeqa: add selftest for python pgo

 meta/lib/oeqa/selftest/cases/python.py             |  28 ++++++
 meta/lib/oeqa/targetcontrol.py                     |  29 +++++-
 meta/recipes-connectivity/iproute2/iproute2.inc    |   7 +-
 ...1-don-t-use-absolute-path-for-ssh-program.patch |  31 ++++++
 meta/recipes-connectivity/openssh/openssh_7.6p1.bb |   7 +-
 meta/recipes-devtools/images/python-pgo-image.bb   |  92 ++++++++++++++++++
 .../python/python-3.5-manifest.inc                 |  12 ++-
 .../python/python3-native_3.5.3.bb                 |   9 ++
 meta/recipes-devtools/python/python3-profile-opt   |   1 +
 .../python/python3-profile-opt_3.5.3.bb            |  10 ++
 .../python3/0001-cross-compile-support.patch       |   9 --
 ...efile-add-install_generate_profile-target.patch |  25 +++++
 ...-CFLAGS-for-extensions-when-cross-compili.patch |  56 +++++++++++
 .../python3-setup.py-no-host-headers-libs.patch    |  33 -------
 ...name-libpython3-to-libpython-profile-opt3.patch | 108 +++++++++++++++++++++
 .../setup.py-find-libraries-in-staging-dirs.patch  |  30 ------
 meta/recipes-devtools/python/python3_3.5.3.bb      |  58 ++++++++---
 ...revent-absolute-path-in-installed-symlink.patch |  29 ++++++
 meta/recipes-extended/iptables/iptables_1.6.1.bb   |   3 +
 meta/recipes-extended/libmnl/libmnl_1.0.4.bb       |   2 +
 scripts/contrib/python/generate-manifest-3.5.py    |   5 +-
 21 files changed, 487 insertions(+), 97 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/python.py
 create mode 100644 meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch
 create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb
 create mode 120000 meta/recipes-devtools/python/python3-profile-opt
 create mode 100644 meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
 create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
 create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
 delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
 create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
 delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
 create mode 100644 meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch

-- 
2.13.6



^ permalink raw reply	[flat|nested] 17+ messages in thread
* [PATCH v8 01/16] python3-native: support profile optimized build
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 02/16] python3: fix depends of python3-tests Markus Lehtonen
                   ` (14 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

Make it possible to build python3-native with profile directed optimization
enabled. The feature is enabled by specifying
PYTHON3_NATIVE_PROFILE_OPT = "1" in local.conf.

The profile task to be run may be specified with PYTHON3_NATIVE_PROFILE_TASK
variable in local.conf, e.g.
PYTHON3_NATIVE_PROFILE_TASK = "${S}/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-devtools/python/python3-native_3.5.3.bb             | 9 +++++++++
 .../python/python3/0001-cross-compile-support.patch              | 9 ---------
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/meta/recipes-devtools/python/python3-native_3.5.3.bb b/meta/recipes-devtools/python/python3-native_3.5.3.bb
index 8cd9c88a82..936ea05b29 100644
--- a/meta/recipes-devtools/python/python3-native_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3-native_3.5.3.bb
@@ -58,11 +58,20 @@ EXTRA_OEMAKE = '\
 # No ctypes option for python 3
 PYTHONLSBOPTS = ""
 
+PYTHON3_NATIVE_MAKE_TARGET ?= "${@'profile-opt' if d.getVar('PYTHON3_NATIVE_PROFILE_OPT', True) == '1' else ''}"
+
 do_configure_append() {
 	autoreconf --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
 	sed -i -e 's,#define HAVE_GETRANDOM 1,/\* #undef HAVE_GETRANDOM \*/,' ${B}/pyconfig.h
 }
 
+do_compile() {
+    if [ -n "${PYTHON3_NATIVE_PROFILE_TASK}" ]; then
+        sed -i -e 's,^PROFILE_TASK=.*,PROFILE_TASK=${PYTHON3_NATIVE_PROFILE_TASK},g' Makefile
+    fi
+    oe_runmake ${PYTHON3_NATIVE_MAKE_TARGET}
+}
+
 do_install() {
 	install -d ${D}${libdir}/pkgconfig
 	oe_runmake 'DESTDIR=${D}' install
diff --git a/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch b/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
index 118d75ddc5..bd599a8b7e 100644
--- a/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
+++ b/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
@@ -34,15 +34,6 @@ index a88b7d5..7cb8bb3 100644
  
  PSRCS=		\
  		Parser/acceler.c \
-@@ -510,7 +512,7 @@ build_all_generate_profile:
- 
- run_profile_task:
- 	: # FIXME: can't run for a cross build
--	$(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true
-+	$(LLVM_PROF_FILE) $(RUNSHARED) $(HOSTPYTHON) $(PROFILE_TASK) || true
- 
- build_all_merge_profile:
- 	$(LLVM_PROF_MERGER)
 @@ -787,7 +789,7 @@ $(IO_OBJS): $(IO_H)
  
  $(GRAMMAR_H): @GENERATED_COMMENT@ $(GRAMMAR_INPUT) $(PGEN)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 02/16] python3: fix depends of python3-tests
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 01/16] python3-native: support profile optimized build Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 03/16] python3: add python3-profile-opt recipe Markus Lehtonen
                   ` (13 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

Similar to an earlier fix for Python 2.7. Make the tests subpackage
depend on all modules as test.regrtest uses most (if not all) of them.

Some tests also depend on libgcc so add that as a runtime dependency as
well.

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-devtools/python/python-3.5-manifest.inc | 2 +-
 scripts/contrib/python/generate-manifest-3.5.py      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
index 0260e87e75..067f6c58c5 100644
--- a/meta/recipes-devtools/python/python-3.5-manifest.inc
+++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
@@ -241,7 +241,7 @@ RDEPENDS_${PN}-terminal="${PN}-core ${PN}-io"
 FILES_${PN}-terminal="${libdir}/python3.5/pty.* ${libdir}/python3.5/__pycache__/pty.* ${libdir}/python3.5/tty.* ${libdir}/python3.5/__pycache__/tty.* "
 
 SUMMARY_${PN}-tests="Python tests"
-RDEPENDS_${PN}-tests="${PN}-core ${PN}-compression"
+RDEPENDS_${PN}-tests="${PN}-core ${PN}-modules libgcc"
 FILES_${PN}-tests="${libdir}/python3.5/test ${libdir}/python3.5/test/__pycache__ "
 
 SUMMARY_${PN}-textutils="Python option parsing, text wrapping and CSV support"
diff --git a/scripts/contrib/python/generate-manifest-3.5.py b/scripts/contrib/python/generate-manifest-3.5.py
index 6352f8f120..1890b91109 100755
--- a/scripts/contrib/python/generate-manifest-3.5.py
+++ b/scripts/contrib/python/generate-manifest-3.5.py
@@ -403,7 +403,7 @@ if __name__ == "__main__":
     m.addPackage( "${PN}-terminal", "Python terminal controlling support", "${PN}-core ${PN}-io",
     "pty.* tty.*" )
 
-    m.addPackage( "${PN}-tests", "Python tests", "${PN}-core ${PN}-compression",
+    m.addPackage( "${PN}-tests", "Python tests", "${PN}-core ${PN}-modules libgcc",
     "test", True ) # package
 
     m.addPackage( "${PN}-threading", "Python threading & synchronization support", "${PN}-core ${PN}-lang",
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 03/16] python3: add python3-profile-opt recipe
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 01/16] python3-native: support profile optimized build Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 02/16] python3: fix depends of python3-tests Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 04/16] python3-profile-opt: rename libpython3 Markus Lehtonen
                   ` (12 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

This patch adds a new recipe that builds a special version of python3 that
produces profile data used for optimization.  The new recipe directly includes
the base python recipe so that they are build in as similar way as possible and
this hopefully decreases the recipe maintenance burden, too. Also, its files
clash with the "normal" python3 so they cannot be installed in an image at the
same time.

Normally, profile-guided-optimization in Python is done simply by doing "make
profile-opt" which first builds python with profile instrumentation enabled,
then runs a profile task to get the profile data, and last, re-builds python
with profile data guiding the optimization. However, in our cross-build
environment this gets a lot trickier. We need to split out the steps as
building is done on the build host but we need to run the second step (i.e. run
the profile task) on the target hardware. This patch enables the first step,
i.e.  building python with profile instrumentation enabled

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-devtools/python/python3-profile-opt   |  1 +
 .../python/python3-profile-opt_3.5.3.bb            |  9 +++++++
 ...efile-add-install_generate_profile-target.patch | 25 ++++++++++++++++++
 meta/recipes-devtools/python/python3_3.5.3.bb      | 30 ++++++++++++----------
 4 files changed, 51 insertions(+), 14 deletions(-)
 create mode 120000 meta/recipes-devtools/python/python3-profile-opt
 create mode 100644 meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
 create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch

diff --git a/meta/recipes-devtools/python/python3-profile-opt b/meta/recipes-devtools/python/python3-profile-opt
new file mode 120000
index 0000000000..b8a0adbbb9
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-profile-opt
@@ -0,0 +1 @@
+python3
\ No newline at end of file
diff --git a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
new file mode 100644
index 0000000000..6940480546
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
@@ -0,0 +1,9 @@
+require python3_${PV}.bb
+
+SRC_URI += "file://Makefile-add-install_generate_profile-target.patch \
+           "
+
+PYTHON3_MAKE_TARGET = "build_all_generate_profile"
+
+RCONFLICTS_${PN}-core = "python3-core"
+RCONFLICTS_lib${BPN} = "libpython3"
diff --git a/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch b/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
new file mode 100644
index 0000000000..7aee7fb799
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
@@ -0,0 +1,25 @@
+Makefile: add install_generate_profile target
+
+Making it possible to install the binaries with profile generation
+enabled. We need this as linking is also performed at the install stage.
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 2b87614..826155b 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1046,6 +1046,8 @@ QUICKTESTOPTS=	$(TESTOPTS) -x test_subprocess test_io test_lib2to3 \
+ quicktest:	all platform
+ 		$(TESTRUNNER) $(QUICKTESTOPTS)
+ 
++install_generate_profile:
++	$(MAKE) install LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
+ 
+ install: @FRAMEWORKINSTALLFIRST@ commoninstall bininstall maninstall @FRAMEWORKINSTALLLAST@
+ 	if test "x$(ENSUREPIP)" != "xno"  ; then \
+-- 
+2.6.6
+
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 281a65f784..bb524eb17d 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -115,14 +115,10 @@ do_compile() {
 	# then call do_install twice we get Makefile.orig == Makefile.sysroot
 	install -m 0644 Makefile Makefile.sysroot
 
-	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
-		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
-		STAGING_LIBDIR=${STAGING_LIBDIR} \
-		STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
-		STAGING_INCDIR=${STAGING_INCDIR} \
-		LIB=${baselib} \
-		ARCH=${TARGET_ARCH} \
-		OPT="${CFLAGS}" libpython3.so
+    if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+        # This is only used in PGO profiling by python3-profile-opt package
+        export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/"
+    fi
 
 	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
 		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
@@ -131,7 +127,7 @@ do_compile() {
 		STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
 		LIB=${baselib} \
 		ARCH=${TARGET_ARCH} \
-		OPT="${CFLAGS}"
+		OPT="${CFLAGS}" ${PYTHON3_MAKE_TARGET}
 }
 
 do_install() {
@@ -151,8 +147,14 @@ do_install() {
 		STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
 		LIB=${baselib} \
 		ARCH=${TARGET_ARCH} \
-		DESTDIR=${D} LIBDIR=${libdir}
+		DESTDIR=${D} LIBDIR=${libdir} ${PYTHON3_MAKE_TARGET}
 	
+    if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+        # Need special make install if pgo generation is enabled
+        _PYTHON3_MAKE_INSTALL_TARGET="install_generate_profile"
+    else
+        _PYTHON3_MAKE_INSTALL_TARGET="install"
+    fi
 	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
 		HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
 		STAGING_LIBDIR=${STAGING_LIBDIR} \
@@ -160,7 +162,7 @@ do_install() {
 		STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
 		LIB=${baselib} \
 		ARCH=${TARGET_ARCH} \
-		DESTDIR=${D} LIBDIR=${libdir} install
+		DESTDIR=${D} LIBDIR=${libdir} ${_PYTHON3_MAKE_INSTALL_TARGET}
 
 	# avoid conflict with 2to3 from Python 2
 	rm -f ${D}/${bindir}/2to3
@@ -209,9 +211,9 @@ PACKAGES =+ "${PN}-pyvenv"
 FILES_${PN}-pyvenv += "${bindir}/pyvenv-${PYTHON_MAJMIN} ${bindir}/pyvenv"
 
 # package libpython3
-PACKAGES =+ "libpython3 libpython3-staticdev"
-FILES_libpython3 = "${libdir}/libpython*.so.*"
-FILES_libpython3-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/libpython${PYTHON_BINABI}.a"
+PACKAGES =+ "lib${BPN} lib${BPN}-staticdev"
+FILES_lib${BPN} = "${libdir}/libpython*.so.*"
+FILES_lib${BPN}-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/libpython*.a"
 INSANE_SKIP_${PN}-dev += "dev-elf"
 
 # catch all the rest (unsorted)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 04/16] python3-profile-opt: rename libpython3
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (2 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 03/16] python3: add python3-profile-opt recipe Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 05/16] openssh: extend to -native Markus Lehtonen
                   ` (11 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

Prevents soname clash with "normal" libpython3. The python library
needs to be renamed so that the automatic ndency generation/checking
mechanism in bitbake does not get confused. Otherwise
python3-profile-opt will depend on libpython3 from the "normal" python3
package.

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 .../python/python3-profile-opt_3.5.3.bb            |   1 +
 ...name-libpython3-to-libpython-profile-opt3.patch | 108 +++++++++++++++++++++
 2 files changed, 109 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch

diff --git a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
index 6940480546..3ad427117f 100644
--- a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
@@ -1,6 +1,7 @@
 require python3_${PV}.bb
 
 SRC_URI += "file://Makefile-add-install_generate_profile-target.patch \
+            file://rename-libpython3-to-libpython-profile-opt3.patch \
            "
 
 PYTHON3_MAKE_TARGET = "build_all_generate_profile"
diff --git a/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch b/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
new file mode 100644
index 0000000000..2e5f9302fc
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
@@ -0,0 +1,108 @@
+rename libpython3 to libpython-profile-opt3
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py
+index 7c278ef..78613d8 100644
+--- a/Lib/distutils/command/build_ext.py
++++ b/Lib/distutils/command/build_ext.py
+@@ -747,7 +747,7 @@ class build_ext(Command):
+         else:
+             from distutils import sysconfig
+             if sysconfig.get_config_var('Py_ENABLE_SHARED'):
+-                pythonlib = 'python{}.{}{}'.format(
++                pythonlib = 'python-profile-opt{}.{}{}'.format(
+                     sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff,
+                     sysconfig.get_config_var('ABIFLAGS'))
+                 return ext.libraries + [pythonlib]
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 0327385..f744ce1 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -623,7 +623,7 @@ $(LIBRARY): $(LIBRARY_OBJS)
+ 	$(AR) $(ARFLAGS) $@ $(MODOBJS)
+ 	$(RANLIB) $@
+ 
+-libpython$(LDVERSION).so: $(LIBRARY_OBJS)
++libpython-profile-opt$(LDVERSION).so: $(LIBRARY_OBJS)
+ 	if test $(INSTSONAME) != $(LDLIBRARY); then \
+ 		$(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ 		$(LN) -f $(INSTSONAME) $@; \
+@@ -631,7 +631,7 @@ libpython$(LDVERSION).so: $(LIBRARY_OBJS)
+ 		$(BLDSHARED) -o $@ $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ 	fi
+ 
+-libpython3.so:	libpython$(LDVERSION).so
++libpython-profile-opt3.so:	libpython-profile-opt$(LDVERSION).so
+ 	$(BLDSHARED) $(NO_AS_NEEDED) -o $@ -Wl,-h$@ $^
+ 
+ libpython$(LDVERSION).dylib: $(LIBRARY_OBJS)
+diff --git a/Misc/python-config.in b/Misc/python-config.in
+index a808f1a..0dcc615 100644
+--- a/Misc/python-config.in
++++ b/Misc/python-config.in
+@@ -47,7 +47,7 @@ for opt in opt_flags:
+         print(' '.join(flags))
+ 
+     elif opt in ('--libs', '--ldflags'):
+-        libs = ['-lpython' + pyver + sys.abiflags]
++        libs = ['-lpython-profile-opt' + pyver + sys.abiflags]
+         libs += getvar('LIBS').split()
+         libs += getvar('SYSLIBS').split()
+         # add the prefix/lib/pythonX.Y/config dir, but only if there is no
+diff --git a/Misc/python-config.sh.in b/Misc/python-config.sh.in
+index 30c6927..efad748 100644
+--- a/Misc/python-config.sh.in
++++ b/Misc/python-config.sh.in
+@@ -40,7 +40,7 @@ LIBM="@LIBM@"
+ LIBC="@LIBC@"
+ SYSLIBS="$LIBM $LIBC"
+ ABIFLAGS="@ABIFLAGS@"
+-LIBS="-lpython${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
++LIBS="-lpython-profile-opt${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
+ BASECFLAGS="@BASECFLAGS@"
+ LDLIBRARY="@LDLIBRARY@"
+ LINKFORSHARED="@LINKFORSHARED@"
+diff --git a/Misc/python.pc.in b/Misc/python.pc.in
+index ae69867..b73d027 100644
+--- a/Misc/python.pc.in
++++ b/Misc/python.pc.in
+@@ -9,5 +9,5 @@ Description: Python library
+ Requires: 
+ Version: @VERSION@
+ Libs.private: @LIBS@
+-Libs: -L${libdir} -lpython@VERSION@@ABIFLAGS@
++Libs: -L${libdir} -lpython-profile-opt@VERSION@@ABIFLAGS@
+ Cflags: -I${includedir}/python@VERSION@@ABIFLAGS@
+diff --git a/configure.ac b/configure.ac
+index d8c2847..ad63e4e 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -1001,7 +1001,7 @@ AC_SUBST(LIBRARY)
+ AC_MSG_CHECKING(LIBRARY)
+ if test -z "$LIBRARY"
+ then
+-	LIBRARY='libpython$(VERSION)$(ABIFLAGS).a'
++	LIBRARY='libpython-profile-opt$(VERSION)$(ABIFLAGS).a'
+ fi
+ AC_MSG_RESULT($LIBRARY)
+ 
+@@ -1162,13 +1162,13 @@ if test $enable_shared = "yes"; then
+ 	  fi
+           ;;
+     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
+-	  LDLIBRARY='libpython$(LDVERSION).so'
+-	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
++	  LDLIBRARY='libpython-profile-opt$(LDVERSION).so'
++	  BLDLIBRARY='-L. -lpython-profile-opt$(LDVERSION)'
+ 	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
+ 	  INSTSONAME="$LDLIBRARY".$SOVERSION
+ 	  if test "$with_pydebug" != yes
+           then
+-	      PY3LIBRARY=libpython3.so
++	      PY3LIBRARY=libpython-profile-opt3.so
+ 	  fi
+ 	  ;;
+     hp*|HP*)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 05/16] openssh: extend to -native
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (3 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 04/16] python3-profile-opt: rename libpython3 Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 06/16] libmnl: enable native Markus Lehtonen
                   ` (10 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 ...1-don-t-use-absolute-path-for-ssh-program.patch | 31 ++++++++++++++++++++++
 meta/recipes-connectivity/openssh/openssh_7.6p1.bb |  7 ++++-
 2 files changed, 37 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch

diff --git a/meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch b/meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch
new file mode 100644
index 0000000000..e9a1c35390
--- /dev/null
+++ b/meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch
@@ -0,0 +1,31 @@
+From fb78bd8142f48d0909d8839e8bab48abe39d5ab7 Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Mon, 20 Nov 2017 12:08:56 +0200
+Subject: [PATCH] don't use absolute path for ssh program
+
+Makes it possible to relocate openssh binaries referencing to the ssh
+binary.
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ Makefile.in | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/Makefile.in b/Makefile.in
+index c52ce19..752f7d0 100644
+--- a/Makefile.in
++++ b/Makefile.in
+@@ -21,7 +21,7 @@ top_srcdir=@top_srcdir@
+ 
+ DESTDIR=
+ VPATH=@srcdir@
+-SSH_PROGRAM=@bindir@/ssh
++SSH_PROGRAM=ssh
+ ASKPASS_PROGRAM=$(libexecdir)/ssh-askpass
+ SFTP_SERVER=$(libexecdir)/sftp-server
+ SSH_KEYSIGN=$(libexecdir)/ssh-keysign
+-- 
+2.13.6
+
diff --git a/meta/recipes-connectivity/openssh/openssh_7.6p1.bb b/meta/recipes-connectivity/openssh/openssh_7.6p1.bb
index ebb9a5734d..b0b9ba3f01 100644
--- a/meta/recipes-connectivity/openssh/openssh_7.6p1.bb
+++ b/meta/recipes-connectivity/openssh/openssh_7.6p1.bb
@@ -27,6 +27,8 @@ SRC_URI = "http://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${PV}.tar
            file://add-test-support-for-busybox.patch \
            "
 
+SRC_URI_append_class-native = " file://0001-don-t-use-absolute-path-for-ssh-program.patch"
+
 PAM_SRC_URI = "file://sshd"
 
 SRC_URI[md5sum] = "06a88699018e5fef13d4655abfed1f63"
@@ -52,7 +54,7 @@ CFLAGS += "-D__FILE_OFFSET_BITS=64"
 EXTRA_OECONF = "'LOGIN_PROGRAM=${base_bindir}/login' \
                 ${@bb.utils.contains('DISTRO_FEATURES', 'pam', '--with-pam', '--without-pam', d)} \
                 --without-zlib-version-check \
-                --with-privsep-path=/var/run/sshd \
+                --with-privsep-path=${localstatedir}/run/sshd \
                 --sysconfdir=${sysconfdir}/ssh \
                 --with-xauth=/usr/bin/xauth \
                 --disable-strip \
@@ -149,6 +151,8 @@ RDEPENDS_${PN} += "${PN}-scp ${PN}-ssh ${PN}-sshd ${PN}-keygen"
 RDEPENDS_${PN}-sshd += "${PN}-keygen ${@bb.utils.contains('DISTRO_FEATURES', 'pam', 'pam-plugin-keyinit pam-plugin-loginuid', '', d)}"
 RDEPENDS_${PN}-ptest += "${PN}-sftp ${PN}-misc ${PN}-sftp-server make"
 
+RDEPENDS_${PN}_class-native = ""
+
 RPROVIDES_${PN}-ssh = "ssh"
 RPROVIDES_${PN}-sshd = "sshd"
 
@@ -163,3 +167,4 @@ ALTERNATIVE_PRIORITY = "90"
 ALTERNATIVE_${PN}-scp = "scp"
 ALTERNATIVE_${PN}-ssh = "ssh"
 
+BBCLASSEXTEND = "native"
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 06/16] libmnl: enable native
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (4 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 05/16] openssh: extend to -native Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 07/16] iptables: " Markus Lehtonen
                   ` (9 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-extended/libmnl/libmnl_1.0.4.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-extended/libmnl/libmnl_1.0.4.bb b/meta/recipes-extended/libmnl/libmnl_1.0.4.bb
index b458799800..23262e8c6f 100644
--- a/meta/recipes-extended/libmnl/libmnl_1.0.4.bb
+++ b/meta/recipes-extended/libmnl/libmnl_1.0.4.bb
@@ -11,3 +11,5 @@ SRC_URI[tar.md5sum] = "be9b4b5328c6da1bda565ac5dffadb2d"
 SRC_URI[tar.sha256sum] = "171f89699f286a5854b72b91d06e8f8e3683064c5901fb09d954a9ab6f551f81"
 
 inherit autotools pkgconfig
+
+BBCLASSEXTEND = "native"
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 07/16] iptables: enable native
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (5 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 06/16] libmnl: enable native Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 08/16] iproute2: " Markus Lehtonen
                   ` (8 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 ...revent-absolute-path-in-installed-symlink.patch | 29 ++++++++++++++++++++++
 meta/recipes-extended/iptables/iptables_1.6.1.bb   |  3 +++
 2 files changed, 32 insertions(+)
 create mode 100644 meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch

diff --git a/meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch b/meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch
new file mode 100644
index 0000000000..3ca6c0dd28
--- /dev/null
+++ b/meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch
@@ -0,0 +1,29 @@
+From d7a50cf5a9f03f3b65293d556dadfc484456b0d8 Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Fri, 17 Nov 2017 14:10:59 +0200
+Subject: [PATCH] prevent absolute path in installed symlink
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ iptables/Makefile.am | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/iptables/Makefile.am b/iptables/Makefile.am
+index f92cc4f..55ce793 100644
+--- a/iptables/Makefile.am
++++ b/iptables/Makefile.am
+@@ -88,7 +88,8 @@ pkgconfig_DATA = xtables.pc
+ install-exec-hook:
+ 	-if test -z "${DESTDIR}"; then /sbin/ldconfig; fi;
+ 	${INSTALL} -dm0755 "${DESTDIR}${bindir}";
+-	for i in ${vx_bin_links}; do ${LN_S} -f "${sbindir}/xtables-multi" "${DESTDIR}${bindir}/$$i"; done;
++	_relpath=`python -c "import os.path; print(os.path.relpath('${sbindir}/xtables-multi', '${bindir}'))"`; \
++	for i in ${vx_bin_links}; do ${LN_S} -f "$${_relpath}" "${DESTDIR}${bindir}/$$i"; done;
+ 	for i in ${v4_sbin_links}; do ${LN_S} -f xtables-multi "${DESTDIR}${sbindir}/$$i"; done;
+ 	for i in ${v6_sbin_links}; do ${LN_S} -f xtables-multi "${DESTDIR}${sbindir}/$$i"; done;
+ 	for i in ${x_sbin_links}; do ${LN_S} -f xtables-compat-multi "${DESTDIR}${sbindir}/$$i"; done;
+-- 
+2.13.6
+
diff --git a/meta/recipes-extended/iptables/iptables_1.6.1.bb b/meta/recipes-extended/iptables/iptables_1.6.1.bb
index b37c55a64e..2f565821a4 100644
--- a/meta/recipes-extended/iptables/iptables_1.6.1.bb
+++ b/meta/recipes-extended/iptables/iptables_1.6.1.bb
@@ -23,6 +23,7 @@ SRC_URI = "http://netfilter.org/projects/iptables/files/iptables-${PV}.tar.bz2 \
            file://0001-configure-Add-option-to-enable-disable-libnfnetlink.patch \
            file://0002-configure.ac-only-check-conntrack-when-libnfnetlink-enabled.patch \
            file://0001-fix-build-with-musl.patch \
+           file://prevent-absolute-path-in-installed-symlink.patch \
 "
 
 SRC_URI[md5sum] = "ab38a33806b6182c6f53d6afb4619add"
@@ -47,3 +48,5 @@ do_configure_prepend() {
 	# Keep ax_check_linker_flags.m4 which belongs to autoconf-archive.
 	rm -f libtool.m4 lt~obsolete.m4 ltoptions.m4 ltsugar.m4 ltversion.m4
 }
+
+BBCLASSEXTEND = "native"
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 08/16] iproute2: enable native
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (6 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 07/16] iptables: " Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 09/16] oeqa/targetcontrol: re-introduce get_target_controller() Markus Lehtonen
                   ` (7 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-connectivity/iproute2/iproute2.inc | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-connectivity/iproute2/iproute2.inc b/meta/recipes-connectivity/iproute2/iproute2.inc
index a578eb3afa..d08d049f41 100644
--- a/meta/recipes-connectivity/iproute2/iproute2.inc
+++ b/meta/recipes-connectivity/iproute2/iproute2.inc
@@ -26,12 +26,15 @@ do_configure_append () {
 
 do_install () {
     oe_runmake DESTDIR=${D} install
-    mv ${D}${base_sbindir}/ip ${D}${base_sbindir}/ip.iproute2
     install -d ${D}${datadir}
     mv ${D}/share/* ${D}${datadir}/ || true
     rm ${D}/share -rf || true
 }
 
+do_install_append_class-target () {
+    mv ${D}${base_sbindir}/ip ${D}${base_sbindir}/ip.iproute2
+}
+
 # The .so files in iproute2-tc are modules, not traditional libraries
 INSANE_SKIP_${PN}-tc = "dev-so"
 
@@ -50,3 +53,5 @@ ALTERNATIVE_${PN} = "ip"
 ALTERNATIVE_TARGET[ip] = "${base_sbindir}/ip.${BPN}"
 ALTERNATIVE_LINK_NAME[ip] = "${base_sbindir}/ip"
 ALTERNATIVE_PRIORITY = "100"
+
+BBCLASSEXTEND = "native"
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 09/16] oeqa/targetcontrol: re-introduce get_target_controller()
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (7 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 08/16] iproute2: " Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 10/16] oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__ Markus Lehtonen
                   ` (6 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

This function was dropped in f352ef37a27cb0871c87cf9991d8ad2e1d560fde as
unused, but, python-pgo-image needs it for running profile task on
target hw.

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/targetcontrol.py | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index f63936c3ec..d147a44480 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -18,6 +18,31 @@ from oeqa.utils.dump import TargetDumper
 from oeqa.controllers.testtargetloader import TestTargetLoader
 from abc import ABCMeta, abstractmethod
 
+def get_target_controller(d, logger):
+    testtarget = d.getVar("TEST_TARGET")
+    # old, simple names
+    if testtarget == "qemu":
+        return QemuTarget(d, logger)
+    elif testtarget == "simpleremote":
+        return SimpleRemoteTarget(d, logger)
+    else:
+        # use the class name
+        try:
+            # is it a core class defined here?
+            controller = getattr(sys.modules[__name__], testtarget)
+        except AttributeError:
+            # nope, perhaps a layer defined one
+            try:
+                bbpath = d.getVar("BBPATH").split(':')
+                testtargetloader = TestTargetLoader()
+                controller = testtargetloader.get_controller_module(testtarget, bbpath)
+            except ImportError as e:
+                bb.fatal("Failed to import {0} from available controller modules:\n{1}".format(testtarget,traceback.format_exc()))
+            except AttributeError as e:
+                bb.fatal("Invalid TEST_TARGET - " + str(e))
+        return controller(d, logger)
+
+
 class BaseTarget(object, metaclass=ABCMeta):
 
     supported_image_fstypes = []
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 10/16] oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (8 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 09/16] oeqa/targetcontrol: re-introduce get_target_controller() Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 11/16] devtools/images: add python-pgo-image Markus Lehtonen
                   ` (5 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/targetcontrol.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index d147a44480..da529e67a4 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -225,8 +225,8 @@ class QemuTarget(BaseTarget):
 
 class SimpleRemoteTarget(BaseTarget):
 
-    def __init__(self, d):
-        super(SimpleRemoteTarget, self).__init__(d)
+    def __init__(self, d, logger):
+        super(SimpleRemoteTarget, self).__init__(d, logger)
         addr = d.getVar("TEST_TARGET_IP") or bb.fatal('Please set TEST_TARGET_IP with the IP address of the machine you want to run the tests on.')
         self.ip = addr.split(":")[0]
         try:
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 11/16] devtools/images: add python-pgo-image
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (9 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 10/16] oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__ Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 12/16] python3: support profile optimized build Markus Lehtonen
                   ` (4 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

This is a special image for profiling Python 3.x in order to utilize
profile-guided-optimization. Profile data can be obtained by by running
bitbake python-pgo-image -c profile. Profile data will be copied into a
directory pointed by PYTHON3_PROFILE_DIR on the host system. The profile
task may be specified with PYTHON3_PROFILE_TASK.

The default profile task is pybench, although an example for using
test.regrtest is included in the recipe.

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-devtools/images/python-pgo-image.bb | 92 ++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb

diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
new file mode 100644
index 0000000000..0dbe8fd052
--- /dev/null
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -0,0 +1,92 @@
+SUMMARY = "Minimal image for doing Python profiling (for PGO)"
+
+IMAGE_FEATURES += "ssh-server-dropbear"
+IMAGE_INSTALL = "packagegroup-core-boot"
+IMAGE_INSTALL += "python3-profile-opt python3-profile-opt-tests python3-profile-opt-tools"
+
+LICENSE = "MIT"
+
+inherit core-image
+
+PROFILE_DATA_WORKDIR = "${WORKDIR}/profile-data"
+
+PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data"
+PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess"
+# Exclude tests that are failing on qemux86
+PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode"
+# Exclude tests that are taking very long on qemux86
+PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec"
+
+# Change default profile target to pybench. Running test.regrtest takes
+# ridiculously long, i.e. around 4 hours in qemux86 on i7-3770K desktop
+# machine. Pybench "only" takes around 55 minutes.
+PYTHON3_PROFILE_TASK_DEFAULT = "${docdir}/python3-profile-opt/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
+
+PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}"
+
+# We need these because we're utilizing the runtime test helpers from oeqa
+TEST_TARGET ?= "qemu"
+TEST_QEMUBOOT_TIMEOUT ?= "1000"
+TEST_LOG_DIR ?= "${WORKDIR}/qemulogs"
+FIND_ROOTFS = "1"
+
+def run_profile(d, profile_bin, profile_task, tgt_in_dir, host_out_dir):
+    import logging
+    from oeqa.targetcontrol import get_target_controller
+    from oeqa.utils import make_logger_bitbake_compatible
+    from oe.utils import getstatusoutput
+
+    logger = make_logger_bitbake_compatible(logging.getLogger("TargetControl"))
+    target = get_target_controller(d, logger)
+    target.deploy()
+    try:
+        # Boot target
+        bootparams = None
+        if d.getVar('VIRTUAL-RUNTIME_init_manager', True) == 'systemd':
+            bootparams = 'systemd.log_level=debug systemd.log_target=console'
+        target.start(extra_bootparams=bootparams)
+
+        # Run profile task
+        ret, output = target.run(profile_bin + ' ' + profile_task, timeout=7200)
+        if ret:
+            bb.fatal("Failed to run profile task on target: %s" % output)
+        ret, output = target.run('tar czf pgo-data.tgz -C %s .' % tgt_in_dir)
+        if ret:
+            bb.fatal("Failed to archive profile data on target: %s" % output)
+
+        # Retrieve and unpack profile data
+        target.copy_from('/home/root/pgo-data.tgz', host_out_dir)
+
+        profile_tarball = os.path.join(host_out_dir, 'pgo-data.tgz')
+        ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, host_out_dir))
+        os.unlink(profile_tarball)
+        if ret:
+            bb.fatal("Failed to unpack python profile data: %s" % output)
+    finally:
+        target.stop()
+
+
+# Profile task for Python 3.x
+python do_profile() {
+    run_profile(d, 'python3',
+                d.getVar('PYTHON3_PROFILE_TASK', True),
+                '/home/root/python3-pgo-profiles/',
+                os.path.join(d.getVar("PROFILE_DATA_WORKDIR", True), 'python3'))
+}
+
+addtask profile after do_build
+
+# Strictly, these are dependencies of the do_profile task, but, we cannot put
+# them in task depends because this would pull all possible build deps into the
+# recipe sysroot causing conflicts (python3 and python3-profile-opt)
+DEPENDS += "qemu-native qemu-helper-native openssh-native iproute2-native"
+do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}/python3"
+
+python do_profile_setscene () {
+    sstate_setscene(d)
+}
+
+SSTATETASKS += "do_profile"
+do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}/python3"
+do_profile[sstate-outputdirs] = "${PYTHON3_PROFILE_DIR}"
+addtask do_profile_setscene
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 12/16] python3: support profile optimized build
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (10 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 11/16] devtools/images: add python-pgo-image Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 13/16] python3: fix profile-optimized build of modules Markus Lehtonen
                   ` (3 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

This patch makes it possible to build python3 with profile directed
optimization. That is, feed python build process with profile data to
guide optimization. This is the third (and the last) step in profile
directed optimization for Python 3.x.

In order to do a profile-optimized build you need to specify
PYTHON3_PROFILE_OPT = "1" in your local.conf, and, have profile data available
in the location pointed to by PYTHON3_PROFILE_DIR. Profile data can be
obtainen e.g. by running bitbake python-pgo-image -c profile3.

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-devtools/python/python3_3.5.3.bb | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index bb524eb17d..844139dfa3 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -79,6 +79,17 @@ export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dy
 # No ctypes option for python 3
 PYTHONLSBOPTS = ""
 
+# Automatic profile guided optimization
+PYTHON3_MAKE_TARGET ?= "${@'build_all_use_profile' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}"
+PYTHON3_PROFILE_DIR ?= "${@'${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}"
+python () {
+    if (d.getVar('PYTHON3_PROFILE_OPT', True) == '1' and
+            d.getVar('PYTHON3_MAKE_TARGET', True) == 'build_all_use_profile'):
+        profile_dir = d.getVar('PYTHON3_PROFILE_DIR', True)
+        bb.utils.mkdirhier(profile_dir)
+        d.setVarFlag('do_compile', 'file-checksums', '%s:True' % profile_dir)
+}
+
 do_configure_append() {
 	rm -f ${S}/Makefile.orig
 	autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -118,6 +129,10 @@ do_compile() {
     if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
         # This is only used in PGO profiling by python3-profile-opt package
         export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/"
+    elif [ -n "${PYTHON3_PROFILE_DIR}" ]; then
+            export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}"
+            # Remove non-optimized build artefacts
+            oe_runmake clean
     fi
 
 	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
@@ -138,6 +153,11 @@ do_install() {
 	install -d ${D}${libdir}/pkgconfig
 	install -d ${D}${libdir}/python${PYTHON_MAJMIN}/config
 
+    # This only has effect if we build with -fprofile-use, e.g. when make
+    # target is build_all_use_profile
+    if [ -n "${PYTHON3_PROFILE_DIR}" ]; then
+        export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}"
+    fi
 	# rerun the build once again with original makefile this time
 	# run install in a separate step to avoid compile/install race
 	oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 13/16] python3: fix profile-optimized build of modules
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (11 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 12/16] python3: support profile optimized build Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 14/16] python3: add python3-tools subpackage Markus Lehtonen
                   ` (2 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

Without this the pgo-related compiler flags are not used in
cross-builds.

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 ...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++++++++++++++++++++++
 meta/recipes-devtools/python/python3_3.5.3.bb      |  1 +
 2 files changed, 57 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch

diff --git a/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch b/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
new file mode 100644
index 0000000000..2e299b3c59
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
@@ -0,0 +1,56 @@
+From d721ebabbf8567d42dfdcd17e170846ba837b04a Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Wed, 23 Nov 2016 16:08:04 +0200
+Subject: [PATCH] Use correct CFLAGS for extensions when cross-compiling
+
+Take PY_CFLAGS_NODIST into account, like in native build. This is needed
+in order to to profile-optimized build. Also, pass EXTRA_CFLAGS to
+profile-optimized build.
+
+Upstream-Status: Pending
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ Makefile.pre.in | 4 ++--
+ setup.py        | 3 ++-
+ 2 files changed, 4 insertions(+), 3 deletions(-)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 4fcaafc..2d20753 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -510,7 +510,7 @@ profile-opt:
+ 	$(MAKE) profile-removal
+ 
+ build_all_generate_profile:
+-	$(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
++	$(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
+ 
+ run_profile_task:
+ 	: # FIXME: can't run for a cross build
+@@ -520,7 +520,7 @@ build_all_merge_profile:
+ 	$(LLVM_PROF_MERGER)
+ 
+ build_all_use_profile:
+-	$(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@"
++	$(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@"
+ 
+ # Compile and run with gcov
+ .PHONY=coverage coverage-lcov coverage-report
+diff --git a/setup.py b/setup.py
+index 7fa9cc3..ef18fe4 100644
+--- a/setup.py
++++ b/setup.py
+@@ -263,7 +263,8 @@ class PyBuildExt(build_ext):
+         # compilers
+         if compiler is not None:
+             if cross_compiling:
+-                (ccshared,cflags) = (os.environ.get('CCSHARED') or '', os.environ.get('CFLAGS') or '')
++                (ccshared,cflags) = (os.environ.get('CCSHARED') or '',
++                                     (os.environ.get('CFLAGS') or '') + ' ' + sysconfig.get_config_var('PY_CFLAGS_NODIST'))
+             else:
+                 (ccshared,cflags) = sysconfig.get_config_vars('CCSHARED','CFLAGS')
+             args['compiler_so'] = compiler + ' ' + ccshared + ' ' + cflags
+-- 
+2.12.3
+
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 844139dfa3..16097e4236 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -40,6 +40,7 @@ SRC_URI += "\
             file://0001-Issue-21272-Use-_sysconfigdata.py-to-initialize-dist.patch \
             file://Fix-29519-weakref-spewing-exceptions-during-interp-f.patch \
             file://pass-missing-libraries-to-Extension-for-mul.patch \
+            file://Use-correct-CFLAGS-for-extensions-when-cross-compili.patch \
            "
 SRC_URI[md5sum] = "57d1f8bfbabf4f2500273fb0706e6f21"
 SRC_URI[sha256sum] = "eefe2ad6575855423ab630f5b51a8ef6e5556f774584c06beab4926f930ddbb0"
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 14/16] python3: add python3-tools subpackage
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (12 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 13/16] python3: fix profile-optimized build of modules Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:06 ` [PATCH v8 15/16] python3: remove two setup.py cross-compile hacks Markus Lehtonen
  2017-11-27 12:07 ` [PATCH v8 16/16] oeqa: add selftest for python pgo Markus Lehtonen
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

Useful in developing Python, e.g. in benchmarking.

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/recipes-devtools/python/python-3.5-manifest.inc | 10 +++++++---
 meta/recipes-devtools/python/python3_3.5.3.bb        |  5 +++++
 scripts/contrib/python/generate-manifest-3.5.py      |  3 +++
 3 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
index 067f6c58c5..db0be9405e 100644
--- a/meta/recipes-devtools/python/python-3.5-manifest.inc
+++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
@@ -4,9 +4,9 @@
 
  
 
-PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
+PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-tools ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
 
-PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules"
+PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-tools ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules"
 
 SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator"
 RDEPENDS_${PN}-2to3="${PN}-core"
@@ -256,6 +256,10 @@ SUMMARY_${PN}-tkinter="Python Tcl/Tk bindings"
 RDEPENDS_${PN}-tkinter="${PN}-core"
 FILES_${PN}-tkinter="${libdir}/python3.5/lib-dynload/_tkinter.*.so ${libdir}/python3.5/lib-dynload/__pycache__/_tkinter.*.so ${libdir}/python3.5/lib-tk ${libdir}/python3.5/lib-tk/__pycache__ ${libdir}/python3.5/tkinter ${libdir}/python3.5/tkinter/__pycache__ "
 
+SUMMARY_${PN}-tools="Useful scripts from Python"
+RDEPENDS_${PN}-tools="${PN}-modules"
+FILES_${PN}-tools="${docdir}/${PN}/Tools "
+
 SUMMARY_${PN}-typing="Python typing support"
 RDEPENDS_${PN}-typing="${PN}-core"
 FILES_${PN}-typing="${libdir}/python3.5/typing.* ${libdir}/python3.5/__pycache__/typing.* "
@@ -277,7 +281,7 @@ RDEPENDS_${PN}-xmlrpc="${PN}-core ${PN}-xml ${PN}-netserver ${PN}-lang ${PN}-pyd
 FILES_${PN}-xmlrpc="${libdir}/python3.5/xmlrpclib.* ${libdir}/python3.5/__pycache__/xmlrpclib.* ${libdir}/python3.5/SimpleXMLRPCServer.* ${libdir}/python3.5/__pycache__/SimpleXMLRPCServer.* ${libdir}/python3.5/DocXMLRPCServer.* ${libdir}/python3.5/__pycache__/DocXMLRPCServer.* ${libdir}/python3.5/xmlrpc ${libdir}/python3.5/xmlrpc/__pycache__ "
 
 SUMMARY_${PN}-modules="All Python modules"
-RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc  "
+RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-tools ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc  "
 ALLOW_EMPTY_${PN}-modules = "1"
 
 
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 16097e4236..7746ffe3b1 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -196,6 +196,11 @@ do_install() {
 	fi
 
 	oe_multilib_header python${PYTHON_BINABI}/pyconfig.h
+
+    # Install tools
+    install -d -m 755 ${D}/${docdir}/${PN}
+    cp -r ${S}/Tools ${D}/${docdir}/${PN}/
+    find ${D}/${docdir}/${PN}/ -name Makefile -delete
 }
 
 do_install_append_class-nativesdk () {
diff --git a/scripts/contrib/python/generate-manifest-3.5.py b/scripts/contrib/python/generate-manifest-3.5.py
index 1890b91109..7621dce1c3 100755
--- a/scripts/contrib/python/generate-manifest-3.5.py
+++ b/scripts/contrib/python/generate-manifest-3.5.py
@@ -430,4 +430,7 @@ if __name__ == "__main__":
     m.addPackage( "${PN}-mailbox", "Python mailbox format support", "${PN}-core ${PN}-mime",
     "mailbox.*" )
 
+    m.addPackage( "${PN}-tools", "Useful scripts from Python", "${PN}-modules",
+    "${docdir}/${PN}/Tools" )
+
     m.make()
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 15/16] python3: remove two setup.py cross-compile hacks
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (13 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 14/16] python3: add python3-tools subpackage Markus Lehtonen
@ 2017-11-27 12:06 ` Markus Lehtonen
  2017-11-27 12:07 ` [PATCH v8 16/16] oeqa: add selftest for python pgo Markus Lehtonen
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:06 UTC (permalink / raw)
  To: openembedded-core

Remove two unneeded hacks. The first hack ("setup.py: no host headers
libs" patch) is not needed because we use cross-compiler (e.g.
i586-oe-linux-gcc) which has not been configured with any host system
include or library directories, and thus, we don't get any host system
directories when running "gcc -E -v".

The second hack becomes useless after the first hack has been removed
and we get the standard include and lib directories normally from gcc.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 .../python3-setup.py-no-host-headers-libs.patch    | 33 ----------------------
 .../setup.py-find-libraries-in-staging-dirs.patch  | 30 --------------------
 meta/recipes-devtools/python/python3_3.5.3.bb      |  2 --
 3 files changed, 65 deletions(-)
 delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
 delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch

diff --git a/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch b/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
deleted file mode 100644
index 2bc8b8c460..0000000000
--- a/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
+++ /dev/null
@@ -1,33 +0,0 @@
-From 43238e1ac13e32984d015c92a5841f3de1fe1d15 Mon Sep 17 00:00:00 2001
-From: Jackie Huang <jackie.huang@windriver.com>
-Date: Tue, 18 Nov 2014 00:07:07 -0500
-Subject: [PATCH] setup.py: no host headers libs
-
-When we are cross-compiling, setup.py should never look in /usr
-or /usr/local to find headers or libraries.
-
-Upstream-Status: Inappropriate [Cross compile specific]
-
-Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
----
- setup.py | 3 ---
- 1 file changed, 3 deletions(-)
-
-diff --git a/setup.py b/setup.py
-index f020b28..e8339cd 100644
---- a/setup.py
-+++ b/setup.py
-@@ -444,10 +444,7 @@ class PyBuildExt(build_ext):
-         if not cross_compiling:
-             add_dir_to_list(self.compiler.library_dirs, os.path.join('/usr/local', sys.lib))
-             add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
--        # only change this for cross builds for 3.3, issues on Mageia
--        if cross_compiling:
-             self.add_gcc_paths()
--        if not cross_compiling:
-             self.add_multiarch_paths()
- 
-         # Add paths specified in the environment variables LDFLAGS and
--- 
-2.0.0
-
diff --git a/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch b/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
deleted file mode 100644
index f26bd38b02..0000000000
--- a/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
+++ /dev/null
@@ -1,30 +0,0 @@
-Patch setup.py so that the detect_modules() function looks for required
-libraries and headers in STAGING_LIBDIR / STAGING_INCDIR.
-
-Without this patch, several extension modules are not built, even though
-their dependencies are present in the compiler's search paths.
-The result is the following warning, and ultimately incomplete packages:
-
-| The necessary bits to build these optional modules were not found:
-| _bz2                  _curses_panel         _dbm
-| _gdbm                 _lzma                 _sqlite3
-| nis                   readline              zlib
-| To find the necessary bits, look in setup.py in detect_modules() for the module's name.
-
-Upstream-Status:  Inappropriate [Cross compile specific]
-
-Signed-off-by: Dominic Sacré <dominic.sacre@gmx.de>
-
-Index: Python-3.4.3/setup.py
-===================================================================
---- Python-3.4.3.orig/setup.py
-+++ Python-3.4.3/setup.py
-@@ -521,6 +521,8 @@ class PyBuildExt(build_ext):
-         else:
-             lib_dirs = self.compiler.library_dirs[:]
-             inc_dirs = self.compiler.include_dirs[:]
-+            lib_dirs.append(os.environ.get('STAGING_LIBDIR'))
-+            inc_dirs.append(os.environ.get('STAGING_INCDIR'))
-         exts = []
-         missing = []
- 
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 7746ffe3b1..3a997a3b4e 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -31,10 +31,8 @@ SRC_URI += "\
             file://unixccompiler.patch \
             file://avoid-ncursesw-include-path.patch \
             file://python3-use-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch \
-            file://python3-setup.py-no-host-headers-libs.patch \
             file://sysconfig.py-add-_PYTHON_PROJECT_SRC.patch \
             file://setup.py-check-cross_compiling-when-get-FLAGS.patch \
-            file://setup.py-find-libraries-in-staging-dirs.patch \
             file://configure.ac-fix-LIBPL.patch \
             file://upstream-random-fixes.patch \
             file://0001-Issue-21272-Use-_sysconfigdata.py-to-initialize-dist.patch \
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
* [PATCH v8 16/16] oeqa: add selftest for python pgo
  2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
                   ` (14 preceding siblings ...)
  2017-11-27 12:06 ` [PATCH v8 15/16] python3: remove two setup.py cross-compile hacks Markus Lehtonen
@ 2017-11-27 12:07 ` Markus Lehtonen
  15 siblings, 0 replies; 17+ messages in thread
From: Markus Lehtonen @ 2017-11-27 12:07 UTC (permalink / raw)
  To: openembedded-core

Check that we're able to run python profile task on qemu and that python
builds when pgo is enabled.

[YOCTO #9338]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/selftest/cases/python.py | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/python.py

diff --git a/meta/lib/oeqa/selftest/cases/python.py b/meta/lib/oeqa/selftest/cases/python.py
new file mode 100644
index 0000000000..3a7edd1b88
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/python.py
@@ -0,0 +1,28 @@
+import os
+import shutil
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_vars, runCmd
+
+
+class PythonTests(OESelftestTestCase):
+
+    def test_python_pgo(self):
+        bbvars = get_bb_vars(['TMPDIR', 'MACHINE'])
+        profile_dir = os.path.join(bbvars['TMPDIR'], 'oeqa', bbvars['MACHINE'], 'python3-pgo-data')
+
+        self.write_config("""
+PYTHON3_PROFILE_OPT = "1"
+PYTHON3_PROFILE_TASK = "-m test.regrtest test_abc"
+PYTHON3_PROFILE_DIR = "{}"
+""".format(profile_dir))
+
+        profile_file = os.path.join(profile_dir, 'Objects', 'object.gcda')
+
+        bitbake('python-pgo-image -cprofile')
+
+        # Check that profile task generated a file
+        self.assertTrue(os.path.isfile(profile_file))
+
+        # Check that python builds with pgo enabled
+        bitbake('python3')
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2017-11-27 12:07 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-27 12:06 [PATCH v8 00/16] support profile-optimized build for Python Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 01/16] python3-native: support profile optimized build Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 02/16] python3: fix depends of python3-tests Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 03/16] python3: add python3-profile-opt recipe Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 04/16] python3-profile-opt: rename libpython3 Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 05/16] openssh: extend to -native Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 06/16] libmnl: enable native Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 07/16] iptables: " Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 08/16] iproute2: " Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 09/16] oeqa/targetcontrol: re-introduce get_target_controller() Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 10/16] oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__ Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 11/16] devtools/images: add python-pgo-image Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 12/16] python3: support profile optimized build Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 13/16] python3: fix profile-optimized build of modules Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 14/16] python3: add python3-tools subpackage Markus Lehtonen
2017-11-27 12:06 ` [PATCH v8 15/16] python3: remove two setup.py cross-compile hacks Markus Lehtonen
2017-11-27 12:07 ` [PATCH v8 16/16] oeqa: add selftest for python pgo Markus Lehtonen

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.