* [PATCH v2 00/20] support profile-optimized build for Python
@ 2017-05-16 13:18 Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 01/20] python-native: support profile optimized build Markus Lehtonen
` (21 more replies)
0 siblings, 22 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
This patchset makes it possible to make a PGO (profile-guided-optimization)
build of python. This version of the patchset is almost identical to v1
submitted back in February, with these changes:
- rebased on top of latest oe-core master
- exclude profile data for Modules/posixmodule of Python 2.7 as it was not
working correctly
[YOCTO #9338]
The following changes since commit 123962018251dfb1d6ca5aa5c0d02534007de3ab:
build-appliance-image: Update to master head revision (2017-05-01 08:56:43 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-9338
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338
Markus Lehtonen (20):
python-native: support profile optimized build
python: add python-profile-opt recipe
python: remove path hack from setup.py
python-profile-opt: rename libpython
devtools/images: add python-pgo-image
python: make profile-optimized build possible
python-pgo-image: exclude tests from the default profile target
python: add python-tools subpackage
python-pgo-image: switch python default profile task to pybench
python-pgo-image: enable sstate for do_profile
python3-native: support profile optimized build
python3: fix depends of python-tests
python3: add python-profile-opt3 recipe
python-pgo-image: profiling for python3
python3: remove two setup.py cross-compile hacks
python3: support profile optimized build
python3: fix profile-optimized build of modules
python-pgo-image: exclude tests from the python3 profile target
python3: add python3-tools subpackage
python-pgo-image: change python3 profile target to pybench
meta/recipes-devtools/images/python-pgo-image.bb | 121 +++++++++++++++++++++
.../python/python-2.7-manifest.inc | 8 +-
.../python/python-3.5-manifest.inc | 10 +-
...x-.so-loading-when-when-running-profile-t.patch | 26 +++++
.../python/python-native_2.7.13.bb | 10 ++
meta/recipes-devtools/python/python-profile-opt3 | 1 +
.../python/python-profile-opt3_3.5.2.bb | 15 +++
.../python/python-profile-opt_2.7.13.bb | 13 +++
.../01-use-proper-tools-for-cross-build.patch | 10 --
...rename-libpython-to-libpython-profile-opt.patch | 94 ++++++++++++++++
.../python/python3-native_3.5.2.bb | 9 ++
.../python/python3/000-cross-compile.patch | 9 --
...efile-add-install_generate_profile-target.patch | 25 +++++
...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++++++++++
.../python3-setup.py-no-host-headers-libs.patch | 33 ------
...name-libpython3-to-libpython-profile-opt3.patch | 111 +++++++++++++++++++
.../setup.py-find-libraries-in-staging-dirs.patch | 30 -----
meta/recipes-devtools/python/python3_3.5.2.bb | 74 +++++++++----
meta/recipes-devtools/python/python_2.7.13.bb | 54 +++++++--
19 files changed, 590 insertions(+), 119 deletions(-)
create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb
create mode 100644 meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch
create mode 120000 meta/recipes-devtools/python/python-profile-opt3
create mode 100644 meta/recipes-devtools/python/python-profile-opt3_3.5.2.bb
create mode 100644 meta/recipes-devtools/python/python-profile-opt_2.7.13.bb
create mode 100644 meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch
create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
--
2.12.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 01/20] python-native: support profile optimized build
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 02/20] python: add python-profile-opt recipe Markus Lehtonen
` (20 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Make it possible to build python-native with profile directed
optimization enabled. The feature is enabled by setting
PYTHON_NATIVE_PROFILE_OPT = "1" in local.conf.
The profile task to run may be specified with PYTHON_NATIVE_PROFILE_TASK
variable in local.conf, e.g.
PYTHON_NATIVE_PROFILE_TASK = "${S}/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
...x-.so-loading-when-when-running-profile-t.patch | 26 ++++++++++++++++++++++
.../python/python-native_2.7.13.bb | 10 +++++++++
2 files changed, 36 insertions(+)
create mode 100644 meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch
diff --git a/meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch b/meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch
new file mode 100644
index 0000000000..e4c89dba03
--- /dev/null
+++ b/meta/recipes-devtools/python/python-native/Makefile-fix-.so-loading-when-when-running-profile-t.patch
@@ -0,0 +1,26 @@
+Makefile: fix .so loading when when running profile task
+
+Use correct search path when doing profiling for profile-optimized
+build. This way we load the correct Python libraries, i.e. the ones from
+our build directory, and not the ones from the host os.
+
+Upstream-Status: Pending
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 2affc41..4355e7f 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -458,7 +458,7 @@ build_all_generate_profile:
+
+ run_profile_task:
+ : # FIXME: can't run for a cross build
+- $(LLVM_PROF_FILE) ./$(BUILDPYTHON) $(PROFILE_TASK) || true
++ $(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true
+
+ build_all_merge_profile:
+ $(LLVM_PROF_MERGER)
+--
+2.6.6
+
diff --git a/meta/recipes-devtools/python/python-native_2.7.13.bb b/meta/recipes-devtools/python/python-native_2.7.13.bb
index 7edf153489..28fb007d32 100644
--- a/meta/recipes-devtools/python/python-native_2.7.13.bb
+++ b/meta/recipes-devtools/python/python-native_2.7.13.bb
@@ -17,6 +17,7 @@ SRC_URI += "\
file://builddir.patch \
file://parallel-makeinst-create-bindir.patch \
file://revert_use_of_sysconfigdata.patch \
+ file://Makefile-fix-.so-loading-when-when-running-profile-t.patch \
"
S = "${WORKDIR}/Python-${PV}"
@@ -35,10 +36,19 @@ EXTRA_OEMAKE = '\
STAGING_INCDIR=${STAGING_INCDIR_NATIVE} \
'
+PYTHON_NATIVE_MAKE_TARGET ?= "${@'profile-opt' if d.getVar('PYTHON_NATIVE_PROFILE_OPT', True) == '1' else ''}"
+
do_configure_append() {
autoreconf --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
}
+do_compile() {
+ if [ -n "${PYTHON_NATIVE_PROFILE_TASK}" ]; then
+ sed -i -e 's,^PROFILE_TASK=.*,PROFILE_TASK=${PYTHON_NATIVE_PROFILE_TASK},g' Makefile
+ fi
+ oe_runmake ${PYTHON_NATIVE_MAKE_TARGET}
+}
+
do_install() {
oe_runmake 'DESTDIR=${D}' install
install -d ${D}${bindir}/${PN}
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 02/20] python: add python-profile-opt recipe
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 01/20] python-native: support profile optimized build Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 03/20] python: remove path hack from setup.py Markus Lehtonen
` (19 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
This patch adds a new recipe that builds a special version of python
that produces profile data for profile guided optimization. It will
install under /opt in order to prevent clash with the "normal" python.
The new recipe directly includes the base python recipe so that they are
build in as similar way as possible and this hopefully decreases the
recipe maintenance burden, too.
Normally, python profile-guided-optimization is done simply by doing
"make profile-opt" which first builds python with profile
instrumentation enabled, then runs a profile task to get the profile
data, and last, re-builds python with profile data guiding the
optimization. However, in our cross-build environment this gets a lot
trickier. We need to split out the steps as building is done on the
build host but we need to run the second step (i.e. run the profile
task) on the target hardware. This patch enables the first step, i.e.
building python with profile instrumentation enabled
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
.../python/python-profile-opt_2.7.13.bb | 17 ++++++++++++++
meta/recipes-devtools/python/python_2.7.13.bb | 26 +++++++++++++++-------
2 files changed, 35 insertions(+), 8 deletions(-)
create mode 100644 meta/recipes-devtools/python/python-profile-opt_2.7.13.bb
diff --git a/meta/recipes-devtools/python/python-profile-opt_2.7.13.bb b/meta/recipes-devtools/python/python-profile-opt_2.7.13.bb
new file mode 100644
index 0000000000..058cc63bc9
--- /dev/null
+++ b/meta/recipes-devtools/python/python-profile-opt_2.7.13.bb
@@ -0,0 +1,17 @@
+require python_${PV}.bb
+
+PROVIDES += "libpython2"
+RPROVIDES_${PN}-core += "libpython2"
+
+# Use special prefix in order to prevent clash with normal python
+STAGING_INCDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/include"
+STAGING_LIBDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/${baselib}"
+TARGET_CFLAGS += "-I${STAGING_INCDIR_DEFAULT}"
+TARGET_CPPFLAGS += "-I${STAGING_INCDIR_DEFAULT}"
+prefix = "/opt"
+exec_prefix = "/opt"
+
+PYTHON_MAKE_TARGET = "build_all_generate_profile"
+
+# Force pkgdata not to clash with python recipe
+PKG_libpython-profile-opt2 = "libpython-profile-opt2.7-1.0"
diff --git a/meta/recipes-devtools/python/python_2.7.13.bb b/meta/recipes-devtools/python/python_2.7.13.bb
index 4ef99523e0..efe02aa8a3 100644
--- a/meta/recipes-devtools/python/python_2.7.13.bb
+++ b/meta/recipes-devtools/python/python_2.7.13.bb
@@ -37,6 +37,11 @@ CONFIGUREOPTS += " --with-system-ffi "
EXTRA_OECONF += "ac_cv_file__dev_ptmx=yes ac_cv_file__dev_ptc=no"
+# These are needed in order to build with modified prefix (python-prorile-opt
+# recipe)
+STAGING_INCDIR_DEFAULT ?= "${STAGING_INCDIR}"
+STAGING_LIBDIR_DEFAULT ?= "${STAGING_LIBDIR}"
+
do_configure_append() {
rm -f ${S}/Makefile.orig
autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -45,12 +50,12 @@ do_configure_append() {
do_compile() {
# regenerate platform specific files, because they depend on system headers
cd ${S}/Lib/plat-linux2
- include=${STAGING_INCDIR} ${STAGING_BINDIR_NATIVE}/python-native/python \
+ include=${STAGING_INCDIR_DEFAULT} ${STAGING_BINDIR_NATIVE}/python-native/python \
${S}/Tools/scripts/h2py.py -i '(u_long)' \
- ${STAGING_INCDIR}/dlfcn.h \
- ${STAGING_INCDIR}/linux/cdrom.h \
- ${STAGING_INCDIR}/netinet/in.h \
- ${STAGING_INCDIR}/sys/types.h
+ ${STAGING_INCDIR_DEFAULT}/dlfcn.h \
+ ${STAGING_INCDIR_DEFAULT}/linux/cdrom.h \
+ ${STAGING_INCDIR_DEFAULT}/netinet/in.h \
+ ${STAGING_INCDIR_DEFAULT}/sys/types.h
sed -e 's,${STAGING_DIR_HOST},,g' -i *.py
cd -
@@ -60,7 +65,7 @@ do_compile() {
if [ ! -f Makefile.orig ]; then
install -m 0644 Makefile Makefile.orig
fi
- sed -i -e 's#^LDFLAGS=.*#LDFLAGS=${LDFLAGS} -L. -L${STAGING_LIBDIR}#g' \
+ sed -i -e 's#^LDFLAGS=.*#LDFLAGS=${LDFLAGS} -L. -L${STAGING_LIBDIR_DEFAULT}#g' \
-e 's,libdir=${libdir},libdir=${STAGING_LIBDIR},g' \
-e 's,libexecdir=${libexecdir},libexecdir=${STAGING_DIR_HOST}${libexecdir},g' \
-e 's,^LIBDIR=.*,LIBDIR=${STAGING_LIBDIR},g' \
@@ -74,13 +79,17 @@ do_compile() {
export CROSS_COMPILE="${TARGET_PREFIX}"
export PYTHONBUILDDIR="${B}"
+ # This is only used in PGO profiling by python-profile-opt package
+ if [ "${PYTHON_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=./python-pgo-profiles/"
+ fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python-native/pgen \
HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python-native/python \
STAGING_LIBDIR=${STAGING_LIBDIR} \
STAGING_INCDIR=${STAGING_INCDIR} \
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
- OPT="${CFLAGS}"
+ OPT="${CFLAGS}" ${PYTHON_MAKE_TARGET}
}
do_install() {
@@ -99,7 +108,8 @@ do_install() {
STAGING_LIBDIR=${STAGING_LIBDIR} \
STAGING_INCDIR=${STAGING_INCDIR} \
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
- DESTDIR=${D} LIBDIR=${libdir}
+ DESTDIR=${D} LIBDIR=${libdir} ${PYTHON_MAKE_TARGET}
+
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python-native/pgen \
HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python-native/python \
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 03/20] python: remove path hack from setup.py
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 01/20] python-native: support profile optimized build Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 02/20] python: add python-profile-opt recipe Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 04/20] python-profile-opt: rename libpython Markus Lehtonen
` (18 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
There should be no need to hack the library and include dirs in
setup.py. The script detects them from compiler output. We need to
remove this is because python-profile-opt recipe is built with a
non-standard prefix setting (i.e. /opt) and the hack breaks the
discovery of system libraries and headers.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
.../python/python/01-use-proper-tools-for-cross-build.patch | 10 ----------
1 file changed, 10 deletions(-)
diff --git a/meta/recipes-devtools/python/python/01-use-proper-tools-for-cross-build.patch b/meta/recipes-devtools/python/python/01-use-proper-tools-for-cross-build.patch
index 366ce3e400..02656d23c5 100644
--- a/meta/recipes-devtools/python/python/01-use-proper-tools-for-cross-build.patch
+++ b/meta/recipes-devtools/python/python/01-use-proper-tools-for-cross-build.patch
@@ -85,13 +85,3 @@ Index: Python-2.7.13/setup.py
assert not self.inplace
basename, tail = os.path.splitext(ext_filename)
newname = basename + "_failed" + tail
-@@ -574,6 +575,9 @@ class PyBuildExt(build_ext):
-
- # XXX Omitted modules: gl, pure, dl, SGI-specific modules
-
-+ lib_dirs = [ os.getenv("STAGING_LIBDIR"), os.getenv("STAGING_BASELIBDIR") ]
-+ inc_dirs = [ os.getenv("STAGING_INCDIR") ]
-+
- #
- # The following modules are all pretty straightforward, and compile
- # on pretty much any POSIXish platform.
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 04/20] python-profile-opt: rename libpython
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (2 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 03/20] python: remove path hack from setup.py Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 05/20] devtools/images: add python-pgo-image Markus Lehtonen
` (17 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
The python library needs to be renamed so that the automatic dependency
generation/checking mechanism in bitbake does not get confused.
Otherwise python-profile-opt will depend on libpython from the "normal"
python package.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
.../python/python-profile-opt_2.7.13.bb | 6 +-
...rename-libpython-to-libpython-profile-opt.patch | 94 ++++++++++++++++++++++
2 files changed, 95 insertions(+), 5 deletions(-)
create mode 100644 meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch
diff --git a/meta/recipes-devtools/python/python-profile-opt_2.7.13.bb b/meta/recipes-devtools/python/python-profile-opt_2.7.13.bb
index 058cc63bc9..df5eee89c8 100644
--- a/meta/recipes-devtools/python/python-profile-opt_2.7.13.bb
+++ b/meta/recipes-devtools/python/python-profile-opt_2.7.13.bb
@@ -1,7 +1,6 @@
require python_${PV}.bb
-PROVIDES += "libpython2"
-RPROVIDES_${PN}-core += "libpython2"
+SRC_URI += "file://rename-libpython-to-libpython-profile-opt.patch"
# Use special prefix in order to prevent clash with normal python
STAGING_INCDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/include"
@@ -12,6 +11,3 @@ prefix = "/opt"
exec_prefix = "/opt"
PYTHON_MAKE_TARGET = "build_all_generate_profile"
-
-# Force pkgdata not to clash with python recipe
-PKG_libpython-profile-opt2 = "libpython-profile-opt2.7-1.0"
diff --git a/meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch b/meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch
new file mode 100644
index 0000000000..06bb4c02b3
--- /dev/null
+++ b/meta/recipes-devtools/python/python/rename-libpython-to-libpython-profile-opt.patch
@@ -0,0 +1,94 @@
+From bc4690886248d87c019cf3c955d5349327f6fdfc Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Mon, 17 Oct 2016 15:58:21 +0300
+Subject: [PATCH] rename libpython to libpython-profile-opt
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ Lib/distutils/command/build_ext.py | 2 +-
+ Makefile.pre.in | 2 +-
+ Misc/python-config.in | 2 +-
+ Misc/python.pc.in | 2 +-
+ configure.ac | 6 +++---
+ 5 files changed, 7 insertions(+), 7 deletions(-)
+
+diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py
+index 3a49454..56cabce 100644
+--- a/Lib/distutils/command/build_ext.py
++++ b/Lib/distutils/command/build_ext.py
+@@ -758,7 +758,7 @@ class build_ext (Command):
+ else:
+ from distutils import sysconfig
+ if sysconfig.get_config_var('Py_ENABLE_SHARED'):
+- template = "python%d.%d"
++ template = "python-profile-opt%d.%d"
+ pythonlib = (template %
+ (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff))
+ return ext.libraries + [pythonlib]
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index d4b81dc..43fe5fe 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -562,7 +562,7 @@ $(LIBRARY): $(LIBRARY_OBJS)
+ $(AR) $(ARFLAGS) $@ $(MODOBJS)
+ $(RANLIB) $@
+
+-libpython$(VERSION).so: $(LIBRARY_OBJS)
++libpython-profile-opt$(VERSION).so: $(LIBRARY_OBJS)
+ if test $(INSTSONAME) != $(LDLIBRARY); then \
+ $(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ $(LN) -f $(INSTSONAME) $@; \
+diff --git a/Misc/python-config.in b/Misc/python-config.in
+index 9ee0fdf..9fb149d 100644
+--- a/Misc/python-config.in
++++ b/Misc/python-config.in
+@@ -44,7 +44,7 @@ for opt in opt_flags:
+ print ' '.join(flags)
+
+ elif opt in ('--libs', '--ldflags'):
+- libs = ['-lpython' + pyver]
++ libs = ['-lpython-profile-opt' + pyver]
+ libs += getvar('LIBS').split()
+ libs += getvar('SYSLIBS').split()
+ # add the prefix/lib/pythonX.Y/config dir, but only if there is no
+diff --git a/Misc/python.pc.in b/Misc/python.pc.in
+index 08481a9..0822e54 100644
+--- a/Misc/python.pc.in
++++ b/Misc/python.pc.in
+@@ -8,6 +8,6 @@ Description: Python library
+ Requires:
+ Version: @VERSION@
+ Libs.private: @LIBS@
+-Libs: -L${libdir} -lpython@VERSION@
++Libs: -L${libdir} -lpython-profile-opt@VERSION@
+ Cflags: -I${includedir}/python@VERSION@
+
+diff --git a/configure.ac b/configure.ac
+index 353b533..0b8e399 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -765,7 +765,7 @@ AC_SUBST(LIBRARY)
+ AC_MSG_CHECKING(LIBRARY)
+ if test -z "$LIBRARY"
+ then
+- LIBRARY='libpython$(VERSION).a'
++ LIBRARY='libpython-profile-opt$(VERSION).a'
+ fi
+ AC_MSG_RESULT($LIBRARY)
+
+@@ -911,8 +911,8 @@ if test $enable_shared = "yes"; then
+ INSTSONAME="$LDLIBRARY".$SOVERSION
+ ;;
+ Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
+- LDLIBRARY='libpython$(VERSION).so'
+- BLDLIBRARY='-L. -lpython$(VERSION)'
++ LDLIBRARY='libpython-profile-opt$(VERSION).so'
++ BLDLIBRARY='-L. -lpython-profile-opt$(VERSION)'
+ RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
+ case $ac_sys_system in
+ FreeBSD*)
+--
+2.6.6
+
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 05/20] devtools/images: add python-pgo-image
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (3 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 04/20] python-profile-opt: rename libpython Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 06/20] python: make profile-optimized build possible Markus Lehtonen
` (16 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
This is a special image for profiling Python in order to utilize
profile-guided-optimization. Profile data can be obtained by by running
bitbake python-pgo-image -c profile. It will be located in the directory
pointed by ${PYTHON_PROFILE_DIR}. The profile task that is run can be
altered by specifying ${PYTHON_PROFILE_TASK}.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 62 ++++++++++++++++++++++++
1 file changed, 62 insertions(+)
create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
new file mode 100644
index 0000000000..af9da20338
--- /dev/null
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -0,0 +1,62 @@
+SUMMARY = "Minimal image for doing Python profiling (for PGO)"
+
+IMAGE_FEATURES += "ssh-server-dropbear"
+IMAGE_INSTALL = "packagegroup-core-boot python-profile-opt python-profile-opt-tests"
+
+LICENSE = "MIT"
+
+inherit core-image
+
+PYTHON_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python/pgo-data"
+PYTHON_PROFILE_TASK ?= "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess"
+
+# We need these because we're utilizing the runtime test helpers from oeqa
+TEST_TARGET ?= "qemu"
+TEST_QEMUBOOT_TIMEOUT ?= "1000"
+TEST_LOG_DIR ?= "${WORKDIR}/qemulogs"
+FIND_ROOTFS = "1"
+
+python do_profile() {
+ from oeqa.targetcontrol import get_target_controller
+ from oe.utils import getstatusoutput
+
+ target = get_target_controller(d)
+ target.deploy()
+ try:
+ # Boot target
+ bootparams = None
+ if d.getVar('VIRTUAL-RUNTIME_init_manager', True) == 'systemd':
+ bootparams = 'systemd.log_level=debug systemd.log_target=console'
+ target.start(extra_bootparams=bootparams)
+
+ # Run profile task
+ profile_cmd = 'LD_LIBRARY_PATH=/opt/lib /opt/bin/python %s' % d.getVar("PYTHON_PROFILE_TASK", True)
+ ret, output = target.run(profile_cmd, timeout=7200)
+ if ret:
+ bb.fatal("Failed to run profile task on target: %s" % output)
+ ret, output = target.run('tar czf pgo-data.tgz -C /home/root/python-pgo-profiles/ .')
+ if ret:
+ bb.fatal("Failed to archive profile data on target: %s" % output)
+
+ # Retrieve and unpack profile data
+ profile_dir = d.getVar("PYTHON_PROFILE_DIR", True)
+ target.copy_from('/home/root/pgo-data.tgz', profile_dir)
+
+ profile_tarball = os.path.join(profile_dir, 'pgo-data.tgz')
+ ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, profile_dir))
+ os.unlink(profile_tarball)
+ if ret:
+ bb.fatal("Failed to unpack python profile data: %s" % output)
+ finally:
+ target.stop()
+
+ # Exclude files causing problems in cross-profiling
+ excludes = [os.path.join(profile_dir, 'Modules', 'posixmodule.gcda')]
+ for path in excludes:
+ if os.path.exists(path):
+ os.unlink(path)
+}
+
+addtask profile after do_build
+do_profile[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot"
+do_profile[cleandirs] = "${PYTHON_PROFILE_DIR}"
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 06/20] python: make profile-optimized build possible
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (4 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 05/20] devtools/images: add python-pgo-image Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 07/20] python-pgo-image: exclude tests from the default profile target Markus Lehtonen
` (15 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
This patch makes it possible to build python with profile directed
optimization. That is, feed python build process with profile data to
guide optimization. This is the third (and the last) step in profile
directed optimization for Python.
In order to do a profile-optimized build you need to set
PYTHON_PROFILE_OPT = "1" in your local.conf. Be sure to get the profile
data for python, preferably by running
"bitbake python-pgo-image -c profile".
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python_2.7.13.bb | 25 ++++++++++++++++++++++++-
1 file changed, 24 insertions(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/python/python_2.7.13.bb b/meta/recipes-devtools/python/python_2.7.13.bb
index efe02aa8a3..d9e458cf72 100644
--- a/meta/recipes-devtools/python/python_2.7.13.bb
+++ b/meta/recipes-devtools/python/python_2.7.13.bb
@@ -42,6 +42,17 @@ EXTRA_OECONF += "ac_cv_file__dev_ptmx=yes ac_cv_file__dev_ptc=no"
STAGING_INCDIR_DEFAULT ?= "${STAGING_INCDIR}"
STAGING_LIBDIR_DEFAULT ?= "${STAGING_LIBDIR}"
+# Automatic profile guided optimization
+PYTHON_MAKE_TARGET ?= "${@'build_all_use_profile' if d.getVar('PYTHON_PROFILE_OPT', True) == '1' else ''}"
+PYTHON_PROFILE_DIR ?= "${@'${TMPDIR}/work-shared/${MACHINE}/python/pgo-data' if d.getVar('PYTHON_PROFILE_OPT', True) == '1' else ''}"
+python () {
+ if (d.getVar('PYTHON_PROFILE_OPT', True) == '1' and
+ d.getVar('PYTHON_MAKE_TARGET', True) == 'build_all_use_profile'):
+ profile_dir = d.getVar('PYTHON_PROFILE_DIR', True)
+ bb.utils.mkdirhier(profile_dir)
+ d.setVarFlag('do_compile', 'file-checksums', '%s:True' % profile_dir)
+}
+
do_configure_append() {
rm -f ${S}/Makefile.orig
autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -79,9 +90,15 @@ do_compile() {
export CROSS_COMPILE="${TARGET_PREFIX}"
export PYTHONBUILDDIR="${B}"
- # This is only used in PGO profiling by python-profile-opt package
if [ "${PYTHON_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+ # This is only used in PGO profiling by python-profile-opt package
export EXTRA_CFLAGS="-fprofile-dir=./python-pgo-profiles/"
+ else
+ if [ -n "${PYTHON_PROFILE_DIR}" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=${PYTHON_PROFILE_DIR}"
+ # Remove non-optimized build artefacts
+ oe_runmake clean
+ fi
fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python-native/pgen \
@@ -99,6 +116,11 @@ do_install() {
export CROSS_COMPILE="${TARGET_PREFIX}"
export PYTHONBUILDDIR="${B}"
+ # This only has effect if we build with -fprofile-use, e.g. when make
+ # target is build_all_use_profile
+ if [ -n "${PYTHON_PROFILE_DIR}" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=${PYTHON_PROFILE_DIR}"
+ fi
# After swizzling the makefile, we need to run the build again.
# install can race with the build so we have to run this first, then install
@@ -146,6 +168,7 @@ py_package_preprocess () {
python -m py_compile ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py
}
+
require python-${PYTHON_MAJMIN}-manifest.inc
# manual dependency additions
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 07/20] python-pgo-image: exclude tests from the default profile target
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (5 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 06/20] python: make profile-optimized build possible Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 08/20] python: add python-tools subpackage Markus Lehtonen
` (14 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Exclue tests that are segfaulting or otherwise failing (on qemux86
target), and thus, causing the profile task to fail.
Also, disable tests that are taking very long to execute. In qemux86
target on my i7-3770K:
test_lib2to3: 1908.8s
test_io: 1278.4s
test_itertools: 436.4s
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
index af9da20338..426d6acb45 100644
--- a/meta/recipes-devtools/images/python-pgo-image.bb
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -8,7 +8,14 @@ LICENSE = "MIT"
inherit core-image
PYTHON_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python/pgo-data"
-PYTHON_PROFILE_TASK ?= "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess"
+PYTHON_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess"
+# Exclude tests that are segfaulting on qemux86 target
+PYTHON_PROFILE_TASK_DEFAULT += "test_bytes test_str test_string test_tuple test_unicode test_userstring test_xmlrpc"
+# Exclude tests that are failing on qemux86
+PYTHON_PROFILE_TASK_DEFAULT += "test_StringIO test_builtin test_calendar test_cmath test_ctypes test_distutils test_exceptions test_getargs test_gzip test_json test_math test_shutil test_socket test_sqlite test_sysconfig test_traceback test_warnings"
+# Exclude tests that are taking very long on qemux86
+PYTHON_PROFILE_TASK_DEFAULT += "test_io test_lib2to3 test_itertools"
+PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}"
# We need these because we're utilizing the runtime test helpers from oeqa
TEST_TARGET ?= "qemu"
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 08/20] python: add python-tools subpackage
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (6 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 07/20] python-pgo-image: exclude tests from the default profile target Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 09/20] python-pgo-image: switch python default profile task to pybench Markus Lehtonen
` (13 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Useful in developing Python, e.g. in benchmarking.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python-2.7-manifest.inc | 8 +++++---
meta/recipes-devtools/python/python_2.7.13.bb | 5 +++++
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/meta/recipes-devtools/python/python-2.7-manifest.inc b/meta/recipes-devtools/python/python-2.7-manifest.inc
index 7ed254bbd6..33189e0f0b 100644
--- a/meta/recipes-devtools/python/python-2.7-manifest.inc
+++ b/meta/recipes-devtools/python/python-2.7-manifest.inc
@@ -4,9 +4,9 @@
-PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib "
+PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib ${PN}-tools"
-PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib ${PN}-modules"
+PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib ${PN}-modules ${PN}-tools"
SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator"
RDEPENDS_${PN}-2to3="${PN}-core"
@@ -284,4 +284,6 @@ SUMMARY_${PN}-modules="All Python modules"
RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-audio ${PN}-bsddb ${PN}-codecs ${PN}-compile ${PN}-compiler ${PN}-compression ${PN}-contextlib ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-fcntl ${PN}-gdbm ${PN}-hotshot ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-plistlib ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-resource ${PN}-robotparser ${PN}-shell ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-zlib "
ALLOW_EMPTY_${PN}-modules = "1"
-
+SUMMARY_${PN}-tools="Useful scripts from Python"
+RDEPENDS_${PN}-tools="${PN}-modules"
+FILES_${PN}-tools="${docdir}/${PN}/Tools"
diff --git a/meta/recipes-devtools/python/python_2.7.13.bb b/meta/recipes-devtools/python/python_2.7.13.bb
index d9e458cf72..c39bcfe447 100644
--- a/meta/recipes-devtools/python/python_2.7.13.bb
+++ b/meta/recipes-devtools/python/python_2.7.13.bb
@@ -148,6 +148,11 @@ do_install() {
fi
oe_multilib_header python${PYTHON_MAJMIN}/pyconfig.h
+
+ # Install tools
+ install -d -m 755 ${D}/${docdir}/${PN}
+ cp -r ${S}/Tools ${D}/${docdir}/${PN}/
+ find ${D}/${docdir}/${PN}/ -name Makefile -delete
}
do_install_append_class-nativesdk () {
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 09/20] python-pgo-image: switch python default profile task to pybench
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (7 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 08/20] python: add python-tools subpackage Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 10/20] python-pgo-image: enable sstate for do_profile Markus Lehtonen
` (12 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Switch to pybench in order to reduce the time taken by profiling.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
index 426d6acb45..33aa9a1fec 100644
--- a/meta/recipes-devtools/images/python-pgo-image.bb
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -1,20 +1,22 @@
SUMMARY = "Minimal image for doing Python profiling (for PGO)"
IMAGE_FEATURES += "ssh-server-dropbear"
-IMAGE_INSTALL = "packagegroup-core-boot python-profile-opt python-profile-opt-tests"
+IMAGE_INSTALL = "packagegroup-core-boot"
+IMAGE_INSTALL += "python-profile-opt python-profile-opt-tests python-profile-opt-tools"
LICENSE = "MIT"
inherit core-image
PYTHON_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python/pgo-data"
-PYTHON_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess"
-# Exclude tests that are segfaulting on qemux86 target
-PYTHON_PROFILE_TASK_DEFAULT += "test_bytes test_str test_string test_tuple test_unicode test_userstring test_xmlrpc"
-# Exclude tests that are failing on qemux86
-PYTHON_PROFILE_TASK_DEFAULT += "test_StringIO test_builtin test_calendar test_cmath test_ctypes test_distutils test_exceptions test_getargs test_gzip test_json test_math test_shutil test_socket test_sqlite test_sysconfig test_traceback test_warnings"
-# Exclude tests that are taking very long on qemux86
-PYTHON_PROFILE_TASK_DEFAULT += "test_io test_lib2to3 test_itertools"
+#PYTHON_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess"
+## Exclude tests that are segfaulting on qemux86 target
+#PYTHON_PROFILE_TASK_DEFAULT += "test_bytes test_str test_string test_tuple test_unicode test_userstring test_xmlrpc"
+## Exclude tests that are failing on qemux86
+#PYTHON_PROFILE_TASK_DEFAULT += "test_StringIO test_builtin test_calendar test_cmath test_ctypes test_distutils test_exceptions test_getargs test_gzip test_json test_math test_shutil test_socket test_sqlite test_sysconfig test_traceback test_warnings"
+## Exclude tests that are taking very long on qemux86
+#PYTHON_PROFILE_TASK_DEFAULT += "test_io test_lib2to3 test_itertools"
+PYTHON_PROFILE_TASK_DEFAULT = "/opt/share/doc/python-profile-opt/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}"
# We need these because we're utilizing the runtime test helpers from oeqa
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 10/20] python-pgo-image: enable sstate for do_profile
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (8 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 09/20] python-pgo-image: switch python default profile task to pybench Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 11/20] python3-native: support profile optimized build Markus Lehtonen
` (11 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
There are caveats however, basically because of the circular dependency
between python-pgo-image and python (python is a build dep of
python-pgo-image, and at the same time, depends on the profile data
generated by python-pgo-image). In practice the sstate data is only used
when no profile data is yet present. Existing profile data (i.e.
non-empty PYTHON_PROFILE_DIR) changes signature hash of python which, in
turn, changes signature of python-pgo-image and causes a sstate
signature mismatch and rebuild of python-pgo-image. Thus, in order to
utilize sstate data the user should run "bitbake python-pgo-image -c clean"
(or otherwise wipe pre-existing profile data) before running
"bitbake python-pgo-image -c profile".
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
index 33aa9a1fec..05252a1a46 100644
--- a/meta/recipes-devtools/images/python-pgo-image.bb
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -9,6 +9,7 @@ LICENSE = "MIT"
inherit core-image
PYTHON_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python/pgo-data"
+PROFILE_DATA_WORKDIR = "${WORKDIR}/profile-data"
#PYTHON_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing test_subprocess"
## Exclude tests that are segfaulting on qemux86 target
#PYTHON_PROFILE_TASK_DEFAULT += "test_bytes test_str test_string test_tuple test_unicode test_userstring test_xmlrpc"
@@ -48,7 +49,7 @@ python do_profile() {
bb.fatal("Failed to archive profile data on target: %s" % output)
# Retrieve and unpack profile data
- profile_dir = d.getVar("PYTHON_PROFILE_DIR", True)
+ profile_dir = d.getVar("PROFILE_DATA_WORKDIR", True)
target.copy_from('/home/root/pgo-data.tgz', profile_dir)
profile_tarball = os.path.join(profile_dir, 'pgo-data.tgz')
@@ -68,4 +69,14 @@ python do_profile() {
addtask profile after do_build
do_profile[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot"
-do_profile[cleandirs] = "${PYTHON_PROFILE_DIR}"
+do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}"
+
+
+python do_profile_setscene () {
+ sstate_setscene(d)
+}
+
+SSTATETASKS += "do_profile"
+do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}"
+do_profile[sstate-outputdirs] = "${PYTHON_PROFILE_DIR}"
+addtask do_profile_setscene
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 11/20] python3-native: support profile optimized build
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (9 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 10/20] python-pgo-image: enable sstate for do_profile Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 12/20] python3: fix depends of python-tests Markus Lehtonen
` (10 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
This feature is virtually identical to that of python-native. Profile
guided optimization is enabled by defining
PYTHON3_NATIVE_PROFILE_OPT = "1"
in local.conf. In addition, the profile task may be defined with
PYTHON3_NATIVE_PROFILE_TASK.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python3-native_3.5.2.bb | 9 +++++++++
meta/recipes-devtools/python/python3/000-cross-compile.patch | 9 ---------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/meta/recipes-devtools/python/python3-native_3.5.2.bb b/meta/recipes-devtools/python/python3-native_3.5.2.bb
index edcf2244f5..1d99e56054 100644
--- a/meta/recipes-devtools/python/python3-native_3.5.2.bb
+++ b/meta/recipes-devtools/python/python3-native_3.5.2.bb
@@ -56,10 +56,19 @@ EXTRA_OEMAKE = '\
# No ctypes option for python 3
PYTHONLSBOPTS = ""
+PYTHON3_NATIVE_MAKE_TARGET ?= "${@'profile-opt' if d.getVar('PYTHON3_NATIVE_PROFILE_OPT', True) == '1' else ''}"
+
do_configure_append() {
autoreconf --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
}
+do_compile() {
+ if [ -n "${PYTHON3_NATIVE_PROFILE_TASK}" ]; then
+ sed -i -e 's,^PROFILE_TASK=.*,PROFILE_TASK=${PYTHON3_NATIVE_PROFILE_TASK},g' Makefile
+ fi
+ oe_runmake ${PYTHON3_NATIVE_MAKE_TARGET}
+}
+
do_install() {
install -d ${D}${libdir}/pkgconfig
oe_runmake 'DESTDIR=${D}' install
diff --git a/meta/recipes-devtools/python/python3/000-cross-compile.patch b/meta/recipes-devtools/python/python3/000-cross-compile.patch
index 2d822218f4..d6d7820ee8 100644
--- a/meta/recipes-devtools/python/python3/000-cross-compile.patch
+++ b/meta/recipes-devtools/python/python3/000-cross-compile.patch
@@ -29,15 +29,6 @@ Index: Python-3.5.2/Makefile.pre.in
PSRCS= \
Parser/acceler.c \
-@@ -509,7 +511,7 @@ build_all_generate_profile:
-
- run_profile_task:
- : # FIXME: can't run for a cross build
-- $(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true
-+ $(LLVM_PROF_FILE) $(RUNSHARED) $(HOSTPYTHON) $(PROFILE_TASK) || true
-
- build_all_merge_profile:
- $(LLVM_PROF_MERGER)
@@ -792,7 +794,7 @@ $(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGEN)
@$(MKDIR_P) Include
# Avoid copying the file onto itself for an in-tree build
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 12/20] python3: fix depends of python-tests
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (10 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 11/20] python3-native: support profile optimized build Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 13/20] python3: add python-profile-opt3 recipe Markus Lehtonen
` (9 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Similar to an earlier fix for Python 2.7. Make the tests subpackage
depend on all modules as test.regrtest uses most (if not all) of them.
Some tests also depend on libgcc so add that as a runtime dependency as
well.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python-3.5-manifest.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
index 1e20f00c64..3f29b53a73 100644
--- a/meta/recipes-devtools/python/python-3.5-manifest.inc
+++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
@@ -241,7 +241,7 @@ RDEPENDS_${PN}-terminal="${PN}-core ${PN}-io"
FILES_${PN}-terminal="${libdir}/python3.5/pty.* ${libdir}/python3.5/__pycache__/pty.* ${libdir}/python3.5/tty.* ${libdir}/python3.5/__pycache__/tty.* "
SUMMARY_${PN}-tests="Python tests"
-RDEPENDS_${PN}-tests="${PN}-core"
+RDEPENDS_${PN}-tests="${PN}-core ${PN}-modules libgcc"
FILES_${PN}-tests="${libdir}/python3.5/test ${libdir}/python3.5/test/__pycache__ "
SUMMARY_${PN}-textutils="Python option parsing, text wrapping and CSV support"
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 13/20] python3: add python-profile-opt3 recipe
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (11 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 12/20] python3: fix depends of python-tests Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 14/20] python-pgo-image: profiling for python3 Markus Lehtonen
` (8 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Similar to python-profile-opt for Python 2.7, this is a special version
of Python3 used for profile guided optimization. It is used for
generating profile data on the target device. Libpython3 is renamed to
libpython-profile-opt3 in order to prevent soname clash with "normal"
libpython3.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python-profile-opt3 | 1 +
.../python/python-profile-opt3_3.5.2.bb | 15 +++
...efile-add-install_generate_profile-target.patch | 25 +++++
...name-libpython3-to-libpython-profile-opt3.patch | 111 +++++++++++++++++++++
meta/recipes-devtools/python/python3_3.5.2.bb | 46 +++++----
5 files changed, 178 insertions(+), 20 deletions(-)
create mode 120000 meta/recipes-devtools/python/python-profile-opt3
create mode 100644 meta/recipes-devtools/python/python-profile-opt3_3.5.2.bb
create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
diff --git a/meta/recipes-devtools/python/python-profile-opt3 b/meta/recipes-devtools/python/python-profile-opt3
new file mode 120000
index 0000000000..b8a0adbbb9
--- /dev/null
+++ b/meta/recipes-devtools/python/python-profile-opt3
@@ -0,0 +1 @@
+python3
\ No newline at end of file
diff --git a/meta/recipes-devtools/python/python-profile-opt3_3.5.2.bb b/meta/recipes-devtools/python/python-profile-opt3_3.5.2.bb
new file mode 100644
index 0000000000..e85807f9f9
--- /dev/null
+++ b/meta/recipes-devtools/python/python-profile-opt3_3.5.2.bb
@@ -0,0 +1,15 @@
+require python3_${PV}.bb
+
+SRC_URI += "file://rename-libpython3-to-libpython-profile-opt3.patch \
+ file://Makefile-add-install_generate_profile-target.patch \
+ "
+
+# Use special prefix in order to prevent clash with the normal python3 package
+STAGING_INCDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/include"
+STAGING_LIBDIR_DEFAULT = "${STAGING_DIR_HOST}/usr/${baselib}"
+TARGET_CFLAGS += "-I${STAGING_INCDIR_DEFAULT}"
+TARGET_CPPFLAGS += "-I${STAGING_INCDIR_DEFAULT}"
+prefix = "/opt"
+exec_prefix = "/opt"
+
+PYTHON3_MAKE_TARGET = "build_all_generate_profile"
diff --git a/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch b/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
new file mode 100644
index 0000000000..7aee7fb799
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
@@ -0,0 +1,25 @@
+Makefile: add install_generate_profile target
+
+Making it possible to install the binaries with profile generation
+enabled. We need this as linking is also performed at the install stage.
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 2b87614..826155b 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1046,6 +1046,8 @@ QUICKTESTOPTS= $(TESTOPTS) -x test_subprocess test_io test_lib2to3 \
+ quicktest: all platform
+ $(TESTRUNNER) $(QUICKTESTOPTS)
+
++install_generate_profile:
++ $(MAKE) install LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
+
+ install: @FRAMEWORKINSTALLFIRST@ commoninstall bininstall maninstall @FRAMEWORKINSTALLLAST@
+ if test "x$(ENSUREPIP)" != "xno" ; then \
+--
+2.6.6
+
diff --git a/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch b/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
new file mode 100644
index 0000000000..4ba553107f
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
@@ -0,0 +1,111 @@
+rename libpython3 to libpython-profile-opt3
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py
+index f03a4e3..4c65aff 100644
+--- a/Lib/distutils/command/build_ext.py
++++ b/Lib/distutils/command/build_ext.py
+@@ -746,7 +746,7 @@ class build_ext(Command):
+ else:
+ from distutils import sysconfig
+ if sysconfig.get_config_var('Py_ENABLE_SHARED'):
+- pythonlib = 'python{}.{}{}'.format(
++ pythonlib = 'python-profile-opt{}.{}{}'.format(
+ sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff,
+ sysconfig.get_config_var('ABIFLAGS'))
+ return ext.libraries + [pythonlib]
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index bace00e..2b87614 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -622,7 +622,7 @@ $(LIBRARY): $(LIBRARY_OBJS)
+ $(AR) $(ARFLAGS) $@ $(MODOBJS)
+ $(RANLIB) $@
+
+-libpython$(LDVERSION).so: $(LIBRARY_OBJS)
++libpython-profile-opt$(LDVERSION).so: $(LIBRARY_OBJS)
+ if test $(INSTSONAME) != $(LDLIBRARY); then \
+ $(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ $(LN) -f $(INSTSONAME) $@; \
+@@ -630,7 +630,7 @@ libpython$(LDVERSION).so: $(LIBRARY_OBJS)
+ $(BLDSHARED) -o $@ $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ fi
+
+-libpython3.so: libpython$(LDVERSION).so
++libpython-profile-opt3.so: libpython-profile-opt$(LDVERSION).so
+ $(BLDSHARED) $(NO_AS_NEEDED) -o $@ -Wl,-h$@ $^
+
+ libpython$(LDVERSION).dylib: $(LIBRARY_OBJS)
+diff --git a/Misc/python-config.in b/Misc/python-config.in
+index a808f1a..0dcc615 100644
+--- a/Misc/python-config.in
++++ b/Misc/python-config.in
+@@ -47,7 +47,7 @@ for opt in opt_flags:
+ print(' '.join(flags))
+
+ elif opt in ('--libs', '--ldflags'):
+- libs = ['-lpython' + pyver + sys.abiflags]
++ libs = ['-lpython-profile-opt' + pyver + sys.abiflags]
+ libs += getvar('LIBS').split()
+ libs += getvar('SYSLIBS').split()
+ # add the prefix/lib/pythonX.Y/config dir, but only if there is no
+diff --git a/Misc/python-config.sh.in b/Misc/python-config.sh.in
+index 30c6927..efad748 100644
+--- a/Misc/python-config.sh.in
++++ b/Misc/python-config.sh.in
+@@ -40,7 +40,7 @@ LIBM="@LIBM@"
+ LIBC="@LIBC@"
+ SYSLIBS="$LIBM $LIBC"
+ ABIFLAGS="@ABIFLAGS@"
+-LIBS="-lpython${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
++LIBS="-lpython-profile-opt${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
+ BASECFLAGS="@BASECFLAGS@"
+ LDLIBRARY="@LDLIBRARY@"
+ LINKFORSHARED="@LINKFORSHARED@"
+diff --git a/Misc/python.pc.in b/Misc/python.pc.in
+index ae69867..b73d027 100644
+--- a/Misc/python.pc.in
++++ b/Misc/python.pc.in
+@@ -9,5 +9,5 @@ Description: Python library
+ Requires:
+ Version: @VERSION@
+ Libs.private: @LIBS@
+-Libs: -L${libdir} -lpython@VERSION@@ABIFLAGS@
++Libs: -L${libdir} -lpython-profile-opt@VERSION@@ABIFLAGS@
+ Cflags: -I${includedir}/python@VERSION@@ABIFLAGS@
+diff --git a/configure.ac b/configure.ac
+index 6bd9322..4bf3dc2 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -990,7 +990,7 @@ AC_SUBST(LIBRARY)
+ AC_MSG_CHECKING(LIBRARY)
+ if test -z "$LIBRARY"
+ then
+- LIBRARY='libpython$(VERSION)$(ABIFLAGS).a'
++ LIBRARY='libpython-profile-opt$(VERSION)$(ABIFLAGS).a'
+ fi
+ AC_MSG_RESULT($LIBRARY)
+
+@@ -1151,13 +1151,13 @@ if test $enable_shared = "yes"; then
+ fi
+ ;;
+ Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
+- LDLIBRARY='libpython$(LDVERSION).so'
+- BLDLIBRARY='-L. -lpython$(LDVERSION)'
++ LDLIBRARY='libpython-profile-opt$(LDVERSION).so'
++ BLDLIBRARY='-L. -lpython-profile-opt$(LDVERSION)'
+ RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
+ INSTSONAME="$LDLIBRARY".$SOVERSION
+ if test "$with_pydebug" != yes
+ then
+- PY3LIBRARY=libpython3.so
++ PY3LIBRARY=libpython-profile-opt3.so
+ fi
+ ;;
+ hp*|HP*)
+--
+2.6.6
+
diff --git a/meta/recipes-devtools/python/python3_3.5.2.bb b/meta/recipes-devtools/python/python3_3.5.2.bb
index 2ff7c9e278..042c2b0339 100644
--- a/meta/recipes-devtools/python/python3_3.5.2.bb
+++ b/meta/recipes-devtools/python/python3_3.5.2.bb
@@ -70,6 +70,10 @@ export _PYTHON_PROJECT_BASE = "${B}"
export _PYTHON_PROJECT_SRC = "${S}"
export CCSHARED = "-fPIC"
+# These enable build with modified prefix (used in python3-prorile-opt recipe)
+STAGING_INCDIR_DEFAULT ?= "${STAGING_INCDIR}"
+STAGING_LIBDIR_DEFAULT ?= "${STAGING_LIBDIR}"
+
# Fix cross compilation of different modules
export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/:${B}/build/lib.linux-${TARGET_ARCH}-${PYTHON_MAJMIN}:${S}/Lib:${S}/Lib/plat-linux"
@@ -84,12 +88,12 @@ do_configure_append() {
do_compile() {
# regenerate platform specific files, because they depend on system headers
cd ${S}/Lib/plat-linux*
- include=${STAGING_INCDIR} ${STAGING_BINDIR_NATIVE}/python3-native/python3 \
+ include=${STAGING_INCDIR_DEFAULT} ${STAGING_BINDIR_NATIVE}/python3-native/python3 \
${S}/Tools/scripts/h2py.py -i '(u_long)' \
- ${STAGING_INCDIR}/dlfcn.h \
- ${STAGING_INCDIR}/linux/cdrom.h \
- ${STAGING_INCDIR}/netinet/in.h \
- ${STAGING_INCDIR}/sys/types.h
+ ${STAGING_INCDIR_DEFAULT}/dlfcn.h \
+ ${STAGING_INCDIR_DEFAULT}/linux/cdrom.h \
+ ${STAGING_INCDIR_DEFAULT}/netinet/in.h \
+ ${STAGING_INCDIR_DEFAULT}/sys/types.h
sed -e 's,${STAGING_DIR_HOST},,g' -i *.py
cd -
@@ -100,7 +104,7 @@ do_compile() {
if [ ! -f Makefile.orig ]; then
install -m 0644 Makefile Makefile.orig
fi
- sed -i -e 's,^CONFIGURE_LDFLAGS=.*,CONFIGURE_LDFLAGS=-L. -L${STAGING_LIBDIR},g' \
+ sed -i -e 's,^CONFIGURE_LDFLAGS=.*,CONFIGURE_LDFLAGS=-L. -L${STAGING_LIBDIR_DEFAULT},g' \
-e 's,libdir=${libdir},libdir=${STAGING_LIBDIR},g' \
-e 's,libexecdir=${libexecdir},libexecdir=${STAGING_DIR_HOST}${libexecdir},g' \
-e 's,^LIBDIR=.*,LIBDIR=${STAGING_LIBDIR},g' \
@@ -112,14 +116,10 @@ do_compile() {
# then call do_install twice we get Makefile.orig == Makefile.sysroot
install -m 0644 Makefile Makefile.sysroot
- oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
- HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
- STAGING_LIBDIR=${STAGING_LIBDIR} \
- STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
- STAGING_INCDIR=${STAGING_INCDIR} \
- LIB=${baselib} \
- ARCH=${TARGET_ARCH} \
- OPT="${CFLAGS}" libpython3.so
+ if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+ # This is only used in PGO profiling by python-profile-opt package
+ export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/"
+ fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
@@ -128,7 +128,7 @@ do_compile() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- OPT="${CFLAGS}"
+ OPT="${CFLAGS}" ${PYTHON3_MAKE_TARGET}
}
do_install() {
@@ -148,8 +148,14 @@ do_install() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- DESTDIR=${D} LIBDIR=${libdir}
+ DESTDIR=${D} LIBDIR=${libdir} ${PYTHON3_MAKE_TARGET}
+ if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+ # Need special make install if pgo generation is enabled
+ _PYTHON3_MAKE_INSTALL_TARGET="install_generate_profile"
+ else
+ _PYTHON3_MAKE_INSTALL_TARGET="install"
+ fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
STAGING_LIBDIR=${STAGING_LIBDIR} \
@@ -157,7 +163,7 @@ do_install() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- DESTDIR=${D} LIBDIR=${libdir} install
+ DESTDIR=${D} LIBDIR=${libdir} ${_PYTHON3_MAKE_INSTALL_TARGET}
# avoid conflict with 2to3 from Python 2
rm -f ${D}/${bindir}/2to3
@@ -206,9 +212,9 @@ PACKAGES =+ "${PN}-pyvenv"
FILES_${PN}-pyvenv += "${bindir}/pyvenv-${PYTHON_MAJMIN} ${bindir}/pyvenv"
# package libpython3
-PACKAGES =+ "libpython3 libpython3-staticdev"
-FILES_libpython3 = "${libdir}/libpython*.so.*"
-FILES_libpython3-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/libpython${PYTHON_BINABI}.a"
+PACKAGES =+ "lib${BPN} lib${BPN}-staticdev"
+FILES_lib${BPN} = "${libdir}/libpython*.so.*"
+FILES_lib${BPN}-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/lib${BPN}*.a"
INSANE_SKIP_${PN}-dev += "dev-elf"
# catch all the rest (unsorted)
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 14/20] python-pgo-image: profiling for python3
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (12 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 13/20] python3: add python-profile-opt3 recipe Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 15/20] python3: remove two setup.py cross-compile hacks Markus Lehtonen
` (7 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Add a new 'do_profile3' task for python-pgo-image that runs profiling
task for python3 and retrieves the profile data, similarly to
'do_profile' for python2. Profile data will be copied into a directory
pointed by PYTHON3_PROFILE_DIR on the host system. The profile task may
be specified with PYTHON3_PROFILE_TASK.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 57 +++++++++++++++++++-----
1 file changed, 45 insertions(+), 12 deletions(-)
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
index 05252a1a46..285e403d32 100644
--- a/meta/recipes-devtools/images/python-pgo-image.bb
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -3,6 +3,7 @@ SUMMARY = "Minimal image for doing Python profiling (for PGO)"
IMAGE_FEATURES += "ssh-server-dropbear"
IMAGE_INSTALL = "packagegroup-core-boot"
IMAGE_INSTALL += "python-profile-opt python-profile-opt-tests python-profile-opt-tools"
+IMAGE_INSTALL += "python-profile-opt3 python-profile-opt3-tests"
LICENSE = "MIT"
@@ -20,13 +21,18 @@ PROFILE_DATA_WORKDIR = "${WORKDIR}/profile-data"
PYTHON_PROFILE_TASK_DEFAULT = "/opt/share/doc/python-profile-opt/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}"
+PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data"
+PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess"
+
+PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}"
+
# We need these because we're utilizing the runtime test helpers from oeqa
TEST_TARGET ?= "qemu"
TEST_QEMUBOOT_TIMEOUT ?= "1000"
TEST_LOG_DIR ?= "${WORKDIR}/qemulogs"
FIND_ROOTFS = "1"
-python do_profile() {
+def run_profile(d, profile_bin, profile_task, tgt_in_dir, host_out_dir):
from oeqa.targetcontrol import get_target_controller
from oe.utils import getstatusoutput
@@ -40,28 +46,34 @@ python do_profile() {
target.start(extra_bootparams=bootparams)
# Run profile task
- profile_cmd = 'LD_LIBRARY_PATH=/opt/lib /opt/bin/python %s' % d.getVar("PYTHON_PROFILE_TASK", True)
- ret, output = target.run(profile_cmd, timeout=7200)
+ ret, output = target.run(profile_bin + ' ' + profile_task, timeout=7200)
if ret:
bb.fatal("Failed to run profile task on target: %s" % output)
- ret, output = target.run('tar czf pgo-data.tgz -C /home/root/python-pgo-profiles/ .')
+ ret, output = target.run('tar czf pgo-data.tgz -C %s .' % tgt_in_dir)
if ret:
bb.fatal("Failed to archive profile data on target: %s" % output)
# Retrieve and unpack profile data
- profile_dir = d.getVar("PROFILE_DATA_WORKDIR", True)
- target.copy_from('/home/root/pgo-data.tgz', profile_dir)
+ target.copy_from('/home/root/pgo-data.tgz', host_out_dir)
- profile_tarball = os.path.join(profile_dir, 'pgo-data.tgz')
- ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, profile_dir))
+ profile_tarball = os.path.join(host_out_dir, 'pgo-data.tgz')
+ ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, host_out_dir))
os.unlink(profile_tarball)
if ret:
bb.fatal("Failed to unpack python profile data: %s" % output)
finally:
target.stop()
+
+# Profile task for Python2
+python do_profile() {
+ outdir = os.path.join(d.getVar("PROFILE_DATA_WORKDIR", True), 'python')
+ run_profile(d, 'LD_LIBRARY_PATH=/opt/lib /opt/bin/python',
+ d.getVar('PYTHON_PROFILE_TASK', True),
+ '/home/root/python-pgo-profiles/',
+ outdir)
# Exclude files causing problems in cross-profiling
- excludes = [os.path.join(profile_dir, 'Modules', 'posixmodule.gcda')]
+ excludes = [os.path.join(outdir, 'Modules', 'posixmodule.gcda')]
for path in excludes:
if os.path.exists(path):
os.unlink(path)
@@ -69,14 +81,35 @@ python do_profile() {
addtask profile after do_build
do_profile[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot"
-do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}"
-
+do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}/python"
python do_profile_setscene () {
sstate_setscene(d)
}
SSTATETASKS += "do_profile"
-do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}"
+do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}/python"
do_profile[sstate-outputdirs] = "${PYTHON_PROFILE_DIR}"
addtask do_profile_setscene
+
+
+# Profile task for Python3
+python do_profile3() {
+ run_profile(d, 'LD_LIBRARY_PATH=/opt/lib /opt/bin/python3',
+ d.getVar('PYTHON3_PROFILE_TASK', True),
+ '/home/root/python3-pgo-profiles/',
+ os.path.join(d.getVar("PROFILE_DATA_WORKDIR", True), 'python3'))
+}
+
+addtask profile3 after do_build
+do_profile3[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot"
+do_profile3[cleandirs] = "${PROFILE_DATA_WORKDIR}/python3"
+
+python do_profile3_setscene () {
+ sstate_setscene(d)
+}
+
+SSTATETASKS += "do_profile3"
+do_profile3[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}/python3"
+do_profile3[sstate-outputdirs] = "${PYTHON3_PROFILE_DIR}"
+addtask do_profile3_setscene
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 15/20] python3: remove two setup.py cross-compile hacks
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (13 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 14/20] python-pgo-image: profiling for python3 Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 16/20] python3: support profile optimized build Markus Lehtonen
` (6 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Remove two unneeded hacks. The first hack ("setup.py: no host headers
libs" patch) is not needed because we use cross-compiler (e.g.
i586-oe-linux-gcc) which has not been configured with any host system
include or library directories, and thus, we don't get any host system
directories when running "gcc -E -v".
The second hack becomes useless after the first hack has been removed
and we get the standard include and lib directories normally from gcc.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
| 33 ----------------------
.../setup.py-find-libraries-in-staging-dirs.patch | 30 --------------------
meta/recipes-devtools/python/python3_3.5.2.bb | 2 --
3 files changed, 65 deletions(-)
delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
diff --git a/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch b/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
deleted file mode 100644
index 2bc8b8c460..0000000000
--- a/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
+++ /dev/null
@@ -1,33 +0,0 @@
-From 43238e1ac13e32984d015c92a5841f3de1fe1d15 Mon Sep 17 00:00:00 2001
-From: Jackie Huang <jackie.huang@windriver.com>
-Date: Tue, 18 Nov 2014 00:07:07 -0500
-Subject: [PATCH] setup.py: no host headers libs
-
-When we are cross-compiling, setup.py should never look in /usr
-or /usr/local to find headers or libraries.
-
-Upstream-Status: Inappropriate [Cross compile specific]
-
-Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
----
- setup.py | 3 ---
- 1 file changed, 3 deletions(-)
-
-diff --git a/setup.py b/setup.py
-index f020b28..e8339cd 100644
---- a/setup.py
-+++ b/setup.py
-@@ -444,10 +444,7 @@ class PyBuildExt(build_ext):
- if not cross_compiling:
- add_dir_to_list(self.compiler.library_dirs, os.path.join('/usr/local', sys.lib))
- add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
-- # only change this for cross builds for 3.3, issues on Mageia
-- if cross_compiling:
- self.add_gcc_paths()
-- if not cross_compiling:
- self.add_multiarch_paths()
-
- # Add paths specified in the environment variables LDFLAGS and
---
-2.0.0
-
diff --git a/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch b/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
deleted file mode 100644
index f26bd38b02..0000000000
--- a/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
+++ /dev/null
@@ -1,30 +0,0 @@
-Patch setup.py so that the detect_modules() function looks for required
-libraries and headers in STAGING_LIBDIR / STAGING_INCDIR.
-
-Without this patch, several extension modules are not built, even though
-their dependencies are present in the compiler's search paths.
-The result is the following warning, and ultimately incomplete packages:
-
-| The necessary bits to build these optional modules were not found:
-| _bz2 _curses_panel _dbm
-| _gdbm _lzma _sqlite3
-| nis readline zlib
-| To find the necessary bits, look in setup.py in detect_modules() for the module's name.
-
-Upstream-Status: Inappropriate [Cross compile specific]
-
-Signed-off-by: Dominic Sacré <dominic.sacre@gmx.de>
-
-Index: Python-3.4.3/setup.py
-===================================================================
---- Python-3.4.3.orig/setup.py
-+++ Python-3.4.3/setup.py
-@@ -521,6 +521,8 @@ class PyBuildExt(build_ext):
- else:
- lib_dirs = self.compiler.library_dirs[:]
- inc_dirs = self.compiler.include_dirs[:]
-+ lib_dirs.append(os.environ.get('STAGING_LIBDIR'))
-+ inc_dirs.append(os.environ.get('STAGING_INCDIR'))
- exts = []
- missing = []
-
diff --git a/meta/recipes-devtools/python/python3_3.5.2.bb b/meta/recipes-devtools/python/python3_3.5.2.bb
index 042c2b0339..18d683e4c7 100644
--- a/meta/recipes-devtools/python/python3_3.5.2.bb
+++ b/meta/recipes-devtools/python/python3_3.5.2.bb
@@ -30,10 +30,8 @@ SRC_URI += "\
file://unixccompiler.patch \
file://avoid-ncursesw-include-path.patch \
file://python3-use-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch \
- file://python3-setup.py-no-host-headers-libs.patch \
file://sysconfig.py-add-_PYTHON_PROJECT_SRC.patch \
file://setup.py-check-cross_compiling-when-get-FLAGS.patch \
- file://setup.py-find-libraries-in-staging-dirs.patch \
file://configure.ac-fix-LIBPL.patch \
file://python3-fix-CVE-2016-1000110.patch \
file://upstream-random-fixes.patch \
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 16/20] python3: support profile optimized build
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (14 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 15/20] python3: remove two setup.py cross-compile hacks Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 17/20] python3: fix profile-optimized build of modules Markus Lehtonen
` (5 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Make it possible to do profile-optimized build. The implementation is
virtually identical to how we do it in Python 2.7. Basically, you need
to define PYTHON3_PROFILE_OPT = "1" in your local.conf and have profile
data available in the location pointed to by PYTHON3_PROFILE_DIR.
Profile data can be obtainen e.g. by running bitbake python-pgo-image -c
profile3.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python3_3.5.2.bb | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/meta/recipes-devtools/python/python3_3.5.2.bb b/meta/recipes-devtools/python/python3_3.5.2.bb
index 18d683e4c7..7a3d2b367f 100644
--- a/meta/recipes-devtools/python/python3_3.5.2.bb
+++ b/meta/recipes-devtools/python/python3_3.5.2.bb
@@ -78,6 +78,17 @@ export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dy
# No ctypes option for python 3
PYTHONLSBOPTS = ""
+# Automatic profile guided optimization
+PYTHON3_MAKE_TARGET ?= "${@'build_all_use_profile' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}"
+PYTHON3_PROFILE_DIR ?= "${@'${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}"
+python () {
+ if (d.getVar('PYTHON3_PROFILE_OPT', True) == '1' and
+ d.getVar('PYTHON3_MAKE_TARGET', True) == 'build_all_use_profile'):
+ profile_dir = d.getVar('PYTHON3_PROFILE_DIR', True)
+ bb.utils.mkdirhier(profile_dir)
+ d.setVarFlag('do_compile', 'file-checksums', '%s:True' % profile_dir)
+}
+
do_configure_append() {
rm -f ${S}/Makefile.orig
autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -117,6 +128,10 @@ do_compile() {
if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
# This is only used in PGO profiling by python-profile-opt package
export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/"
+ elif [ -n "${PYTHON3_PROFILE_DIR}" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}"
+ # Remove non-optimized build artefacts
+ oe_runmake clean
fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
@@ -137,6 +152,11 @@ do_install() {
install -d ${D}${libdir}/pkgconfig
install -d ${D}${libdir}/python${PYTHON_MAJMIN}/config
+ # This only has effect if we build with -fprofile-use, e.g. when make
+ # target is build_all_use_profile
+ if [ -n "${PYTHON3_PROFILE_DIR}" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}"
+ fi
# rerun the build once again with original makefile this time
# run install in a separate step to avoid compile/install race
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 17/20] python3: fix profile-optimized build of modules
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (15 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 16/20] python3: support profile optimized build Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 18/20] python-pgo-image: exclude tests from the python3 profile target Markus Lehtonen
` (4 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Without this the pgo-related compiler flags are not used in
cross-builds.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++++++++++++++++++++++
meta/recipes-devtools/python/python3_3.5.2.bb | 1 +
2 files changed, 57 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
diff --git a/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch b/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
new file mode 100644
index 0000000000..bf239c6261
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
@@ -0,0 +1,56 @@
+From 0fd8b986888ddf1995b503edf46ac827e83114d9 Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Wed, 23 Nov 2016 16:08:04 +0200
+Subject: [PATCH] Use correct CFLAGS for extensions when cross-compiling
+
+Take PY_CFLAGS_NODIST into account, like in native build. This is needed
+in order to to profile-optimized build. Also, pass EXTRA_CFLAGS to
+profile-optimized build.
+
+Upstream-Status: Pending
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ Makefile.pre.in | 4 ++--
+ setup.py | 3 ++-
+ 2 files changed, 4 insertions(+), 3 deletions(-)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index bace00e..9635aa4 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -509,7 +509,7 @@ profile-opt:
+ $(MAKE) profile-removal
+
+ build_all_generate_profile:
+- $(MAKE) all CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
++ $(MAKE) all CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
+
+ run_profile_task:
+ : # FIXME: can't run for a cross build
+@@ -519,7 +519,7 @@ build_all_merge_profile:
+ $(LLVM_PROF_MERGER)
+
+ build_all_use_profile:
+- $(MAKE) all CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@"
++ $(MAKE) all CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@"
+
+ # Compile and run with gcov
+ .PHONY=coverage coverage-lcov coverage-report
+diff --git a/setup.py b/setup.py
+index c1ce87e..72d37cf 100644
+--- a/setup.py
++++ b/setup.py
+@@ -271,7 +271,8 @@ class PyBuildExt(build_ext):
+ # compilers
+ if compiler is not None:
+ if cross_compiling:
+- (ccshared,cflags) = (os.environ.get('CCSHARED') or '', os.environ.get('CFLAGS') or '')
++ (ccshared,cflags) = (os.environ.get('CCSHARED') or '',
++ (os.environ.get('CFLAGS') or '') + ' ' + sysconfig.get_config_var('PY_CFLAGS_NODIST'))
+ else:
+ (ccshared,cflags) = sysconfig.get_config_vars('CCSHARED','CFLAGS')
+ args['compiler_so'] = compiler + ' ' + ccshared + ' ' + cflags
+--
+2.6.6
+
diff --git a/meta/recipes-devtools/python/python3_3.5.2.bb b/meta/recipes-devtools/python/python3_3.5.2.bb
index 7a3d2b367f..7853c69e09 100644
--- a/meta/recipes-devtools/python/python3_3.5.2.bb
+++ b/meta/recipes-devtools/python/python3_3.5.2.bb
@@ -35,6 +35,7 @@ SRC_URI += "\
file://configure.ac-fix-LIBPL.patch \
file://python3-fix-CVE-2016-1000110.patch \
file://upstream-random-fixes.patch \
+ file://Use-correct-CFLAGS-for-extensions-when-cross-compili.patch \
"
SRC_URI[md5sum] = "8906efbacfcdc7c3c9198aeefafd159e"
SRC_URI[sha256sum] = "0010f56100b9b74259ebcd5d4b295a32324b58b517403a10d1a2aa7cb22bca40"
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 18/20] python-pgo-image: exclude tests from the python3 profile target
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (16 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 17/20] python3: fix profile-optimized build of modules Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 19/20] python3: add python3-tools subpackage Markus Lehtonen
` (3 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Exlude failing tests (on qemux86 target) which would cause the profile
task to fail.
Also, disable some very long running tests. Running in qemux86 target on
my i7-3770K:
test_lib2to3: 1604.5s
test_buffer: 1562.8s
test_pickle: 1527.0s
test_io: 1517.8s
test_threading: 1384.1s
test_asyncio: 1380.9s
test_urllib2_localnet: 1276.8s
test_itertools: 944.7s
test_tuple: 847.7s
test_trace: 756.7s
test_tarfile: 716.7s
test_unicodedata: 709.9s
test_decimal: 615.1s
test_buffer: 1562.8s
test_pickle: 1527.0s
test_io: 1517.8s
test_threading: 1384.1s
test_asyncio: 1380.9s
test_urllib2_localnet: 1276.8s
test_itertools: 944.7s
test_tuple: 847.7s
test_trace: 756.7s
test_tarfile: 716.7s
test_unicodedata: 709.9s
test_decimal: 615.1s
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
index 285e403d32..e16cd52486 100644
--- a/meta/recipes-devtools/images/python-pgo-image.bb
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -23,6 +23,11 @@ PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}"
PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data"
PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess"
+# Exclude tests that are failing on qemux86
+PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode"
+# Exclude tests that are taking very long on qemux86
+PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal"
+PYTHON3_PROFILE_TASK_DEFAULT += "test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec"
PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}"
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 19/20] python3: add python3-tools subpackage
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (17 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 18/20] python-pgo-image: exclude tests from the python3 profile target Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 20/20] python-pgo-image: change python3 profile target to pybench Markus Lehtonen
` (2 subsequent siblings)
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python-3.5-manifest.inc | 8 +++++---
meta/recipes-devtools/python/python3_3.5.2.bb | 5 +++++
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
index 3f29b53a73..8e40aec9c9 100644
--- a/meta/recipes-devtools/python/python-3.5-manifest.inc
+++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
@@ -4,9 +4,9 @@
-PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
+PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-tools "
-PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules"
+PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules ${PN}-tools"
SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator"
RDEPENDS_${PN}-2to3="${PN}-core"
@@ -280,4 +280,6 @@ SUMMARY_${PN}-modules="All Python modules"
RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
ALLOW_EMPTY_${PN}-modules = "1"
-
+SUMMARY_${PN}-tools="Useful scripts from Python"
+RDEPENDS_${PN}-tools="${PN}-modules"
+FILES_${PN}-tools="${docdir}/${PN}/Tools"
diff --git a/meta/recipes-devtools/python/python3_3.5.2.bb b/meta/recipes-devtools/python/python3_3.5.2.bb
index 7853c69e09..ef1a9b4da7 100644
--- a/meta/recipes-devtools/python/python3_3.5.2.bb
+++ b/meta/recipes-devtools/python/python3_3.5.2.bb
@@ -195,6 +195,11 @@ do_install() {
fi
oe_multilib_header python${PYTHON_BINABI}/pyconfig.h
+
+ # Install tools
+ install -d -m 755 ${D}/${docdir}/${PN}
+ cp -r ${S}/Tools ${D}/${docdir}/${PN}/
+ find ${D}/${docdir}/${PN}/ -name Makefile -delete
}
do_install_append_class-nativesdk () {
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 20/20] python-pgo-image: change python3 profile target to pybench
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (18 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 19/20] python3: add python3-tools subpackage Markus Lehtonen
@ 2017-05-16 13:18 ` Markus Lehtonen
2017-05-16 13:23 ` [PATCH v2 00/20] support profile-optimized build for Python Alexander Kanavin
2017-05-16 13:45 ` Richard Purdie
21 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 13:18 UTC (permalink / raw)
To: openembedded-core
Running test.regrtest was still taking ridiculously long, i.e. around 4
hours in qemux86 on my desktop machine. Change to pybench which "only"
takes around 55 minutes.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
index e16cd52486..dc98e0c839 100644
--- a/meta/recipes-devtools/images/python-pgo-image.bb
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -3,7 +3,7 @@ SUMMARY = "Minimal image for doing Python profiling (for PGO)"
IMAGE_FEATURES += "ssh-server-dropbear"
IMAGE_INSTALL = "packagegroup-core-boot"
IMAGE_INSTALL += "python-profile-opt python-profile-opt-tests python-profile-opt-tools"
-IMAGE_INSTALL += "python-profile-opt3 python-profile-opt3-tests"
+IMAGE_INSTALL += "python-profile-opt3 python-profile-opt3-tests python-profile-opt3-tools"
LICENSE = "MIT"
@@ -22,12 +22,13 @@ PYTHON_PROFILE_TASK_DEFAULT = "/opt/share/doc/python-profile-opt/Tools/pybench/p
PYTHON_PROFILE_TASK ?= "${PYTHON_PROFILE_TASK_DEFAULT}"
PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data"
-PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess"
+#PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess"
# Exclude tests that are failing on qemux86
-PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode"
+#PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode"
# Exclude tests that are taking very long on qemux86
-PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal"
-PYTHON3_PROFILE_TASK_DEFAULT += "test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec"
+#PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal"
+#PYTHON3_PROFILE_TASK_DEFAULT += "test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec"
+PYTHON3_PROFILE_TASK_DEFAULT = "/opt/share/doc/python-profile-opt3/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}"
--
2.12.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH v2 00/20] support profile-optimized build for Python
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (19 preceding siblings ...)
2017-05-16 13:18 ` [PATCH v2 20/20] python-pgo-image: change python3 profile target to pybench Markus Lehtonen
@ 2017-05-16 13:23 ` Alexander Kanavin
2017-05-16 14:07 ` Markus Lehtonen
2017-05-16 13:45 ` Richard Purdie
21 siblings, 1 reply; 26+ messages in thread
From: Alexander Kanavin @ 2017-05-16 13:23 UTC (permalink / raw)
To: Markus Lehtonen, openembedded-core
On 05/16/2017 04:18 PM, Markus Lehtonen wrote:
> This patchset makes it possible to make a PGO (profile-guided-optimization)
> build of python. This version of the patchset is almost identical to v1
> submitted back in February, with these changes:
> - rebased on top of latest oe-core master
> - exclude profile data for Modules/posixmodule of Python 2.7 as it was not
> working correctly
How about doing this only for Python 3.x? We'd like to stick to our
resolution of deprecating 2.x, which means not introducing any new
dependencies or images that need it.
Also, python-pgo-image commits should probably be all squashed into one
(with comments within files replacing the commit messages).
Alex
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH v2 00/20] support profile-optimized build for Python
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
` (20 preceding siblings ...)
2017-05-16 13:23 ` [PATCH v2 00/20] support profile-optimized build for Python Alexander Kanavin
@ 2017-05-16 13:45 ` Richard Purdie
2017-05-16 14:16 ` Markus Lehtonen
21 siblings, 1 reply; 26+ messages in thread
From: Richard Purdie @ 2017-05-16 13:45 UTC (permalink / raw)
To: Markus Lehtonen, openembedded-core
On Tue, 2017-05-16 at 16:18 +0300, Markus Lehtonen wrote:
> This patchset makes it possible to make a PGO (profile-guided-
> optimization) build of python. This version of the patchset is almost
> identical to v1 submitted back in February, with these changes:
> - rebased on top of latest oe-core master - exclude profile data for
> Modules/posixmodule of Python 2.7 as it was not working correctly
>
> [YOCTO #9338]
>
I'm wondering if recipe specific sysroots might have made this problem
a bit easier?
It should now be possible to build the two packages with conflicting
files and as long as you exclude the recipe from the shlibs code and
set the RPROVIDES correctly, it should all work without as many
invasive changes?
Of course I haven't tested that...
Cheers,
Richard
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH v2 00/20] support profile-optimized build for Python
2017-05-16 13:23 ` [PATCH v2 00/20] support profile-optimized build for Python Alexander Kanavin
@ 2017-05-16 14:07 ` Markus Lehtonen
2017-05-16 14:08 ` Alexander Kanavin
0 siblings, 1 reply; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 14:07 UTC (permalink / raw)
To: Alexander Kanavin, openembedded-core
On 16/05/2017, 16.23, "Alexander Kanavin" <alexander.kanavin@linux.intel.com> wrote:
On 05/16/2017 04:18 PM, Markus Lehtonen wrote:
> This patchset makes it possible to make a PGO (profile-guided-optimization)
> build of python. This version of the patchset is almost identical to v1
> submitted back in February, with these changes:
> - rebased on top of latest oe-core master
> - exclude profile data for Modules/posixmodule of Python 2.7 as it was not
> working correctly
How about doing this only for Python 3.x? We'd like to stick to our
resolution of deprecating 2.x, which means not introducing any new
dependencies or images that need it.
You're probably right. There's not much use in churning soon-to-be-deprecated Python 2.x. I will submit a new patchset for Python 3.x only. Python 2.x could be added later if deprecation wouldn't happen for some reason.
Also, python-pgo-image commits should probably be all squashed into one
(with comments within files replacing the commit messages).
Point taken.
Thanks,
Markus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH v2 00/20] support profile-optimized build for Python
2017-05-16 14:07 ` Markus Lehtonen
@ 2017-05-16 14:08 ` Alexander Kanavin
0 siblings, 0 replies; 26+ messages in thread
From: Alexander Kanavin @ 2017-05-16 14:08 UTC (permalink / raw)
To: Markus Lehtonen, openembedded-core
On 05/16/2017 05:07 PM, Markus Lehtonen wrote:
> You're probably right. There's not much use in churning
> soon-to-be-deprecated Python 2.x. I will submit a new patchset for
> Python 3.x only. Python 2.x could be added later if deprecation
> wouldn't happen for some reason.
The deprecation is not a single point in time, it's an ongoing process.
A lot of work was done last spring, and some more work has been done
this spring. The situation with what still needs py 2.x will be
revisited again in a year's time. I just want to avoid regressions on
this front :)
Alex
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH v2 00/20] support profile-optimized build for Python
2017-05-16 13:45 ` Richard Purdie
@ 2017-05-16 14:16 ` Markus Lehtonen
0 siblings, 0 replies; 26+ messages in thread
From: Markus Lehtonen @ 2017-05-16 14:16 UTC (permalink / raw)
To: Richard Purdie, openembedded-core
On 16/05/2017, 16.45, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:
On Tue, 2017-05-16 at 16:18 +0300, Markus Lehtonen wrote:
> This patchset makes it possible to make a PGO (profile-guided-
> optimization) build of python. This version of the patchset is almost
> identical to v1 submitted back in February, with these changes:
> - rebased on top of latest oe-core master - exclude profile data for
> Modules/posixmodule of Python 2.7 as it was not working correctly
>
> [YOCTO #9338]
>
I'm wondering if recipe specific sysroots might have made this problem
a bit easier?
It should now be possible to build the two packages with conflicting
files and as long as you exclude the recipe from the shlibs code and
set the RPROVIDES correctly, it should all work without as many
invasive changes?
Of course I haven't tested that...
I'll try that out. That, accompanied with dropping Python 2.x support (thanks Alexander) would make the patchset significantly simpler.
Cheers,
MArkus
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2017-05-16 14:16 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-16 13:18 [PATCH v2 00/20] support profile-optimized build for Python Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 01/20] python-native: support profile optimized build Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 02/20] python: add python-profile-opt recipe Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 03/20] python: remove path hack from setup.py Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 04/20] python-profile-opt: rename libpython Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 05/20] devtools/images: add python-pgo-image Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 06/20] python: make profile-optimized build possible Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 07/20] python-pgo-image: exclude tests from the default profile target Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 08/20] python: add python-tools subpackage Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 09/20] python-pgo-image: switch python default profile task to pybench Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 10/20] python-pgo-image: enable sstate for do_profile Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 11/20] python3-native: support profile optimized build Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 12/20] python3: fix depends of python-tests Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 13/20] python3: add python-profile-opt3 recipe Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 14/20] python-pgo-image: profiling for python3 Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 15/20] python3: remove two setup.py cross-compile hacks Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 16/20] python3: support profile optimized build Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 17/20] python3: fix profile-optimized build of modules Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 18/20] python-pgo-image: exclude tests from the python3 profile target Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 19/20] python3: add python3-tools subpackage Markus Lehtonen
2017-05-16 13:18 ` [PATCH v2 20/20] python-pgo-image: change python3 profile target to pybench Markus Lehtonen
2017-05-16 13:23 ` [PATCH v2 00/20] support profile-optimized build for Python Alexander Kanavin
2017-05-16 14:07 ` Markus Lehtonen
2017-05-16 14:08 ` Alexander Kanavin
2017-05-16 13:45 ` Richard Purdie
2017-05-16 14:16 ` Markus Lehtonen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.