[PATCH 1/4] scripts/install-buildtools: improvements

[PATCH 1/4] scripts/install-buildtools: improvements

[Tim Orling]

* Install directory defaults to scripts/../buildtools
  e.g. --directory is set by default
  This avoids the user having to type in their sudo password
  to install in /opt/poky/<installer-version>

* Use "." rather than "source" for sourcing the environment script
  as not all distros (e.g. Debian) have "source" by default.

* Add buildtools/ to .gitignore

* Fix typos in example usage (--install-version -> --installer-version)

[YOCTO #13832]

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
---
 .gitignore                 |  1 +
 scripts/install-buildtools | 13 +++++++++----
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/.gitignore b/.gitignore
index d0e6b2fb89..b66d371aac 100644
--- a/.gitignore
+++ b/.gitignore
@@ -9,6 +9,7 @@ pstage/
 scripts/oe-git-proxy-socks
 sources/
 meta-*/
+buildtools/
 !meta-skeleton
 !meta-selftest
 hob-image-*.bb
diff --git a/scripts/install-buildtools b/scripts/install-buildtools
index 0947e9c4d6..49cab1345a 100755
--- a/scripts/install-buildtools
+++ b/scripts/install-buildtools
@@ -17,7 +17,7 @@
 #        $ install-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto \
 #          --release yocto-3.1_M2 \
-#          --install-version 3.0+snapshot
+#          --installer-version 3.0+snapshot
 #          --build-date 202000122
 #
 #  Example usage (standard buildtools from release):
@@ -29,7 +29,7 @@
 #        $ install-buildtools --without-extended-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto \
 #          --release yocto-3.0.2 \
-#          --install-version 3.0.2
+#          --installer-version 3.0.2
 #
 
 import argparse
@@ -59,6 +59,7 @@ if not bitbakepath:
 PROGNAME = 'install-buildtools'
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
+DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools')
 DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto'
 DEFAULT_RELEASE: str = 'yocto-3.1_M2'
 DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
 
 
 def main():
+    global DEFAULT_INSTALL_DIR
     global DEFAULT_BASE_URL
     global DEFAULT_RELEASE
     global DEFAULT_INSTALLER_VERSION
@@ -73,6 +75,7 @@ def main():
     filename: str = ""
     release: str = ""
     buildtools_url: str = ""
+    install_dir: str = ""
 
     parser = argparse.ArgumentParser(
         description="Buildtools installation helper",
@@ -87,6 +90,7 @@ def main():
                              '(optional)\nRequires --url',
                         action='store')
     parser.add_argument('-d', '--directory',
+                        default=DEFAULT_INSTALL_DIR,
                         help='directory where buildtools SDK will be installed (optional)',
                         action='store')
     parser.add_argument('-r', '--release',
@@ -216,12 +220,12 @@ def main():
         st = os.stat(tmpbuildtools)
         os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
         logger.debug(os.stat(tmpbuildtools))
-        install_dir = "/opt/poky/%s" % args.installer_version
         if args.directory:
             install_dir = args.directory
             ret = subprocess.call("%s -d %s -y" %
                                   (tmpbuildtools, install_dir), shell=True)
         else:
+            install_dir = "/opt/poky/%s" % args.installer_version
             ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
         if ret != 0:
             logger.error("Could not run buildtools installer")
@@ -238,7 +242,8 @@ def main():
             tool = 'gcc'
         else:
             tool = 'tar'
-        proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" %
+        logger.debug("install_dir: %s" % install_dir)
+        proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" %
                               (install_dir, tool),
                               shell=True, stdout=subprocess.PIPE)
         which_tool = proc.stdout.decode("utf-8")
-- 
2.24.0

[PATCH 2/4] oe-buildenv-internal: python 3.5 as min version

[Tim Orling]

Python 3.4 is EOL:
https://www.python.org/downloads/release/python-3410/

The last supported distro was probably CentOS-7, which has python36 available
from epel-7 or scl (as rh-python36) [1]

[1] https://www.softwarecollections.org/en/scls/rhscl/rh-python36/

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
---
 scripts/oe-buildenv-internal | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
index 8cbe34669d..c62688fbd2 100755
--- a/scripts/oe-buildenv-internal
+++ b/scripts/oe-buildenv-internal
@@ -32,12 +32,12 @@ fi
 # We potentially have code that doesn't parse correctly with older versions 
 # of Python, and rather than fixing that and being eternally vigilant for 
 # any other new feature use, just check the version here.
-py_v34_check=$(python3 -c 'import sys; print(sys.version_info >= (3,4,0))')
-if [ "$py_v34_check" != "True" ]; then
-    echo >&2 "BitBake requires Python 3.4.0 or later as 'python3'"
+py_v35_check=$(python3 -c 'import sys; print(sys.version_info >= (3,5,0))')
+if [ "$py_v35_check" != "True" ]; then
+    echo >&2 "BitBake requires Python 3.5.0 or later as 'python3'"
     return 1
 fi
-unset py_v34_check
+unset py_v35_check
 
 if [ -z "$BDIR" ]; then
     if [ -z "$1" ]; then
-- 
2.24.0

[PATCH 3/4] sanity.bbclass: recommend using install-buildtools

[Tim Orling]

For old tar versions (1.24 and 1.28), recommned using
scripts/install-buildtools

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
---
 meta/classes/sanity.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index cca5cdad17..0cf955a341 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -533,9 +533,9 @@ def check_tar_version(sanity_data):
         return "Unable to execute tar --version, exit code %d\n%s\n" % (e.returncode, e.output)
     version = result.split()[3]
     if LooseVersion(version) < LooseVersion("1.24"):
-        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+).\n"
+        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n"
     if LooseVersion(version) < LooseVersion("1.28"):
-        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release).\n"
+        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n"
     return None
 
 # We use git parameters and functionality only found in 1.7.8 or later
-- 
2.24.0

[PATCH 4/4] sanity.bbclass: add test for gcc < 5.0

[Tim Orling]

It is known that the version of gcc in CentOS-7 (4.8.5) causes builds to fail.

Add a test for BUILD_CC == 'gcc' and gcc < 5.0 and recommend using
scripts/install-buildtools or user built buildtools-extended-tarball.

NOTE: another solution is to install devtoolset-6+ from scl [1], but
this is a rather large install (> 1 Gb) and fairly invasive.

[1] https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
---
 meta/classes/sanity.bbclass | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 0cf955a341..a1cb8c6b88 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -521,6 +521,29 @@ def check_wsl(d):
             return "OpenEmbedded doesn't work under WSL at this time, sorry"
     return None
 
+# The gcc version in CentOS-7 (4.8.5) is known to be a problem.
+# Require at least gcc version 5.0.
+#
+# This can be fixed on CentOS-7 with devtoolset-6+
+# https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/
+#
+# A less invasive fix is with scripts/install-buildtools (or with user
+# built buildtools-extended-tarball)
+#
+def check_gcc_version(sanity_data):
+    from distutils.version import LooseVersion
+    import subprocess
+    build_cc = sanity_data.getVar('BUILD_CC').strip()
+    if build_cc == "gcc":
+        try:
+            result = subprocess.check_output(["gcc", "--version"], stderr=subprocess.DEVNULL).decode('utf-8')
+        except subprocess.CalledProcessError as e:
+            return "Unable to execute gcc --version, exit code %d\n%s\n" % (e.returncode, e.output)
+        version = result.split()[2]
+        if LooseVersion(version) < LooseVersion("5.0"):
+            return "Your version of gcc is older than 5.0 and will break builds. Please install a newer version of gcc (you could use the project's buildtools-extended-tarball or use scripts/install-buildtools).\n"
+    return None
+
 # Tar version 1.24 and onwards handle overwriting symlinks correctly
 # but earlier versions do not; this needs to work properly for sstate
 # Version 1.28 is needed so opkg-build works correctly when reproducibile builds are enabled
@@ -535,7 +558,7 @@ def check_tar_version(sanity_data):
     if LooseVersion(version) < LooseVersion("1.24"):
         return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n"
     if LooseVersion(version) < LooseVersion("1.28"):
-        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n"
+        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the project's buildtools-tarball from our last release or use scripts/install-buildtools).\n"
     return None
 
 # We use git parameters and functionality only found in 1.7.8 or later
@@ -634,6 +657,7 @@ def check_sanity_version_change(status, d):
     except ImportError as e:
         status.addresult('Your Python 3 is not a full install. Please install the module %s (see the Getting Started guide for further information).\n' % e.name)
 
+    status.addresult(check_gcc_version(d))
     status.addresult(check_make_version(d))
     status.addresult(check_patch_version(d))
     status.addresult(check_tar_version(d))
-- 
2.24.0

Re: [OE-core] [PATCH 3/4] sanity.bbclass: recommend using install-buildtools

[Andre McCurdy]

On Mon, Mar 30, 2020 at 12:43 PM Tim Orling
<timothy.t.orling@linux.intel.com> wrote:
>
> For old tar versions (1.24 and 1.28), recommned using

Typo.

> scripts/install-buildtools
>
> Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
> ---
>  meta/classes/sanity.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index cca5cdad17..0cf955a341 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -533,9 +533,9 @@ def check_tar_version(sanity_data):
>          return "Unable to execute tar --version, exit code %d\n%s\n" % (e.returncode, e.output)
>      version = result.split()[3]
>      if LooseVersion(version) < LooseVersion("1.24"):
> -        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+).\n"
> +        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n"
>      if LooseVersion(version) < LooseVersion("1.28"):
> -        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release).\n"
> +        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n"
>      return None

Is it useful to keep two levels of version check?

>  # We use git parameters and functionality only found in 1.7.8 or later
> --
> 2.24.0
>
> 

Re: [OE-core] [PATCH 4/4] sanity.bbclass: add test for gcc < 5.0

[Richard Purdie]

On Mon, 2020-03-30 at 12:43 -0700, Tim Orling wrote:
> +# The gcc version in CentOS-7 (4.8.5) is known to be a problem.
> +# Require at least gcc version 5.0.
> +#
> +# This can be fixed on CentOS-7 with devtoolset-6+
> +# https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/
> +#
> +# A less invasive fix is with scripts/install-buildtools (or with user
> +# built buildtools-extended-tarball)
> +#
> +def check_gcc_version(sanity_data):
> +    from distutils.version import LooseVersion
> +    import subprocess
> +    build_cc = sanity_data.getVar('BUILD_CC').strip()
> +    if build_cc == "gcc":
> +        try:
> +            result = subprocess.check_output(["gcc", "--version"], stderr=subprocess.DEVNULL).decode('utf-8')
> +        except subprocess.CalledProcessError as e:
> +            return "Unable to execute gcc --version, exit code %d\n%s\n" % (e.returncode, e.output)
> +        version = result.split()[2]
> +        if LooseVersion(version) < LooseVersion("5.0"):
> +            return "Your version of gcc is older than 5.0 and will break builds. Please install a newer version of gcc (you could use the project's buildtools-extended-tarball or use scripts/install-buildtools).\n"
> +    return None

I had forgotten we had this but a recent patch reminded me, would we
want to tweak this function:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oe/utils.py?h=master-next#n376

?

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] scripts/install-buildtools: improvements

[Andre McCurdy]

On Mon, Mar 30, 2020 at 12:43 PM Tim Orling
<timothy.t.orling@linux.intel.com> wrote:
>
> * Install directory defaults to scripts/../buildtools
>   e.g. --directory is set by default
>   This avoids the user having to type in their sudo password
>   to install in /opt/poky/<installer-version>
>
> * Use "." rather than "source" for sourcing the environment script
>   as not all distros (e.g. Debian) have "source" by default.
>
> * Add buildtools/ to .gitignore
>
> * Fix typos in example usage (--install-version -> --installer-version)
>
> [YOCTO #13832]
>
> Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
> ---
>  .gitignore                 |  1 +
>  scripts/install-buildtools | 13 +++++++++----
>  2 files changed, 10 insertions(+), 4 deletions(-)
>
> diff --git a/.gitignore b/.gitignore
> index d0e6b2fb89..b66d371aac 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -9,6 +9,7 @@ pstage/
>  scripts/oe-git-proxy-socks
>  sources/
>  meta-*/
> +buildtools/
>  !meta-skeleton
>  !meta-selftest
>  hob-image-*.bb
> diff --git a/scripts/install-buildtools b/scripts/install-buildtools
> index 0947e9c4d6..49cab1345a 100755
> --- a/scripts/install-buildtools
> +++ b/scripts/install-buildtools
> @@ -17,7 +17,7 @@
>  #        $ install-buildtools \
>  #          --base-url http://downloads.yoctoproject.org/releases/yocto \
>  #          --release yocto-3.1_M2 \
> -#          --install-version 3.0+snapshot
> +#          --installer-version 3.0+snapshot
>  #          --build-date 202000122
>  #
>  #  Example usage (standard buildtools from release):
> @@ -29,7 +29,7 @@
>  #        $ install-buildtools --without-extended-buildtools \
>  #          --base-url http://downloads.yoctoproject.org/releases/yocto \
>  #          --release yocto-3.0.2 \
> -#          --install-version 3.0.2
> +#          --installer-version 3.0.2
>  #
>
>  import argparse
> @@ -59,6 +59,7 @@ if not bitbakepath:
>  PROGNAME = 'install-buildtools'
>  logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
>
> +DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools')

It looks like there are some assumptions about the minimum version of
python3 on the host in order to even be able to run
install-buildtools. Should that be checked explicitly?

>  DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto'
>  DEFAULT_RELEASE: str = 'yocto-3.1_M2'
>  DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
> @@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
>
>
>  def main():
> +    global DEFAULT_INSTALL_DIR
>      global DEFAULT_BASE_URL
>      global DEFAULT_RELEASE
>      global DEFAULT_INSTALLER_VERSION
> @@ -73,6 +75,7 @@ def main():
>      filename: str = ""
>      release: str = ""
>      buildtools_url: str = ""
> +    install_dir: str = ""
>
>      parser = argparse.ArgumentParser(
>          description="Buildtools installation helper",
> @@ -87,6 +90,7 @@ def main():
>                               '(optional)\nRequires --url',
>                          action='store')
>      parser.add_argument('-d', '--directory',
> +                        default=DEFAULT_INSTALL_DIR,
>                          help='directory where buildtools SDK will be installed (optional)',
>                          action='store')
>      parser.add_argument('-r', '--release',
> @@ -216,12 +220,12 @@ def main():
>          st = os.stat(tmpbuildtools)
>          os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
>          logger.debug(os.stat(tmpbuildtools))
> -        install_dir = "/opt/poky/%s" % args.installer_version
>          if args.directory:
>              install_dir = args.directory
>              ret = subprocess.call("%s -d %s -y" %
>                                    (tmpbuildtools, install_dir), shell=True)
>          else:
> +            install_dir = "/opt/poky/%s" % args.installer_version
>              ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
>          if ret != 0:
>              logger.error("Could not run buildtools installer")
> @@ -238,7 +242,8 @@ def main():
>              tool = 'gcc'
>          else:
>              tool = 'tar'
> -        proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" %
> +        logger.debug("install_dir: %s" % install_dir)
> +        proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" %
>                                (install_dir, tool),
>                                shell=True, stdout=subprocess.PIPE)
>          which_tool = proc.stdout.decode("utf-8")
> --
> 2.24.0
>
> 

Re: [OE-core] [PATCH 1/4] scripts/install-buildtools: improvements

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 5603 bytes --]

On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <armccurdy@gmail.com> wrote:

> On Mon, Mar 30, 2020 at 12:43 PM Tim Orling
> <timothy.t.orling@linux.intel.com> wrote:
> >
> > * Install directory defaults to scripts/../buildtools
> >   e.g. --directory is set by default
> >   This avoids the user having to type in their sudo password
> >   to install in /opt/poky/<installer-version>
> >
> > * Use "." rather than "source" for sourcing the environment script
> >   as not all distros (e.g. Debian) have "source" by default.
> >
> > * Add buildtools/ to .gitignore
> >
> > * Fix typos in example usage (--install-version -> --installer-version)
> >
> > [YOCTO #13832]
> >
> > Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
> > ---
> >  .gitignore                 |  1 +
> >  scripts/install-buildtools | 13 +++++++++----
> >  2 files changed, 10 insertions(+), 4 deletions(-)
> >
> > diff --git a/.gitignore b/.gitignore
> > index d0e6b2fb89..b66d371aac 100644
> > --- a/.gitignore
> > +++ b/.gitignore
> > @@ -9,6 +9,7 @@ pstage/
> >  scripts/oe-git-proxy-socks
> >  sources/
> >  meta-*/
> > +buildtools/
> >  !meta-skeleton
> >  !meta-selftest
> >  hob-image-*.bb
> > diff --git a/scripts/install-buildtools b/scripts/install-buildtools
> > index 0947e9c4d6..49cab1345a 100755
> > --- a/scripts/install-buildtools
> > +++ b/scripts/install-buildtools
> > @@ -17,7 +17,7 @@
> >  #        $ install-buildtools \
> >  #          --base-url http://downloads.yoctoproject.org/releases/yocto
> \
> >  #          --release yocto-3.1_M2 \
> > -#          --install-version 3.0+snapshot
> > +#          --installer-version 3.0+snapshot
> >  #          --build-date 202000122
> >  #
> >  #  Example usage (standard buildtools from release):
> > @@ -29,7 +29,7 @@
> >  #        $ install-buildtools --without-extended-buildtools \
> >  #          --base-url http://downloads.yoctoproject.org/releases/yocto
> \
> >  #          --release yocto-3.0.2 \
> > -#          --install-version 3.0.2
> > +#          --installer-version 3.0.2
> >  #
> >
> >  import argparse
> > @@ -59,6 +59,7 @@ if not bitbakepath:
> >  PROGNAME = 'install-buildtools'
> >  logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
> >
> > +DEFAULT_INSTALL_DIR: str =
> os.path.join(os.path.split(scripts_path)[0],'buildtools')
>
> It looks like there are some assumptions about the minimum version of
> python3 on the host in order to even be able to run
> install-buildtools. Should that be checked explicitly?
>

To be honest, everything was written on CentOS-7, where ‘python3’ is now
python36.

The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also
happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete
mess.

Python 3.4 is already EOL. We should not support it for the complete build
system, but I can see that the install-buildtools script should ideally
still run on python34. The whole point is to make it easier to install the
pre-built buildtools.

Debian-8 is EOL in June 2020.


> >  DEFAULT_BASE_URL: str = '
> http://downloads.yoctoproject.org/releases/yocto'
> >  DEFAULT_RELEASE: str = 'yocto-3.1_M2'
> >  DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
> > @@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
> >
> >
> >  def main():
> > +    global DEFAULT_INSTALL_DIR
> >      global DEFAULT_BASE_URL
> >      global DEFAULT_RELEASE
> >      global DEFAULT_INSTALLER_VERSION
> > @@ -73,6 +75,7 @@ def main():
> >      filename: str = ""
> >      release: str = ""
> >      buildtools_url: str = ""
> > +    install_dir: str = ""
> >
> >      parser = argparse.ArgumentParser(
> >          description="Buildtools installation helper",
> > @@ -87,6 +90,7 @@ def main():
> >                               '(optional)\nRequires --url',
> >                          action='store')
> >      parser.add_argument('-d', '--directory',
> > +                        default=DEFAULT_INSTALL_DIR,
> >                          help='directory where buildtools SDK will be
> installed (optional)',
> >                          action='store')
> >      parser.add_argument('-r', '--release',
> > @@ -216,12 +220,12 @@ def main():
> >          st = os.stat(tmpbuildtools)
> >          os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
> >          logger.debug(os.stat(tmpbuildtools))
> > -        install_dir = "/opt/poky/%s" % args.installer_version
> >          if args.directory:
> >              install_dir = args.directory
> >              ret = subprocess.call("%s -d %s -y" %
> >                                    (tmpbuildtools, install_dir),
> shell=True)
> >          else:
> > +            install_dir = "/opt/poky/%s" % args.installer_version
> >              ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
> >          if ret != 0:
> >              logger.error("Could not run buildtools installer")
> > @@ -238,7 +242,8 @@ def main():
> >              tool = 'gcc'
> >          else:
> >              tool = 'tar'
> > -        proc = subprocess.run("source
> %s/environment-setup-x86_64-pokysdk-linux && which %s" %
> > +        logger.debug("install_dir: %s" % install_dir)
> > +        proc = subprocess.run(".
> %s/environment-setup-x86_64-pokysdk-linux && which %s" %
> >                                (install_dir, tool),
> >                                shell=True, stdout=subprocess.PIPE)
> >          which_tool = proc.stdout.decode("utf-8")
> > --
> > 2.24.0
> >
> >
> 
>

[-- Attachment #2: Type: text/html, Size: 7861 bytes --]

Re: [OE-core] [PATCH 1/4] scripts/install-buildtools: improvements

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 6148 bytes --]

On Wed, Apr 1, 2020 at 2:32 PM Tim Orling via lists.openembedded.org
<ticotimo=gmail.com@lists.openembedded.org> wrote:

>
>
> On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>
>> On Mon, Mar 30, 2020 at 12:43 PM Tim Orling
>> <timothy.t.orling@linux.intel.com> wrote:
>> >
>> > * Install directory defaults to scripts/../buildtools
>> >   e.g. --directory is set by default
>> >   This avoids the user having to type in their sudo password
>> >   to install in /opt/poky/<installer-version>
>> >
>> > * Use "." rather than "source" for sourcing the environment script
>> >   as not all distros (e.g. Debian) have "source" by default.
>> >
>> > * Add buildtools/ to .gitignore
>> >
>> > * Fix typos in example usage (--install-version -> --installer-version)
>> >
>> > [YOCTO #13832]
>> >
>> > Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
>> > ---
>> >  .gitignore                 |  1 +
>> >  scripts/install-buildtools | 13 +++++++++----
>> >  2 files changed, 10 insertions(+), 4 deletions(-)
>> >
>> > diff --git a/.gitignore b/.gitignore
>> > index d0e6b2fb89..b66d371aac 100644
>> > --- a/.gitignore
>> > +++ b/.gitignore
>> > @@ -9,6 +9,7 @@ pstage/
>> >  scripts/oe-git-proxy-socks
>> >  sources/
>> >  meta-*/
>> > +buildtools/
>> >  !meta-skeleton
>> >  !meta-selftest
>> >  hob-image-*.bb
>> > diff --git a/scripts/install-buildtools b/scripts/install-buildtools
>> > index 0947e9c4d6..49cab1345a 100755
>> > --- a/scripts/install-buildtools
>> > +++ b/scripts/install-buildtools
>> > @@ -17,7 +17,7 @@
>> >  #        $ install-buildtools \
>> >  #          --base-url http://downloads.yoctoproject.org/releases/yocto
>> \
>> >  #          --release yocto-3.1_M2 \
>> > -#          --install-version 3.0+snapshot
>> > +#          --installer-version 3.0+snapshot
>> >  #          --build-date 202000122
>> >  #
>> >  #  Example usage (standard buildtools from release):
>> > @@ -29,7 +29,7 @@
>> >  #        $ install-buildtools --without-extended-buildtools \
>> >  #          --base-url http://downloads.yoctoproject.org/releases/yocto
>> \
>> >  #          --release yocto-3.0.2 \
>> > -#          --install-version 3.0.2
>> > +#          --installer-version 3.0.2
>> >  #
>> >
>> >  import argparse
>> > @@ -59,6 +59,7 @@ if not bitbakepath:
>> >  PROGNAME = 'install-buildtools'
>> >  logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
>> >
>> > +DEFAULT_INSTALL_DIR: str =
>> os.path.join(os.path.split(scripts_path)[0],'buildtools')
>>
>> It looks like there are some assumptions about the minimum version of
>> python3 on the host in order to even be able to run
>> install-buildtools. Should that be checked explicitly?
>>
>
> To be honest, everything was written on CentOS-7, where ‘python3’ is now
> python36.
>
> The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also
> happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete
> mess.
>
> Python 3.4 is already EOL. We should not support it for the complete build
> system, but I can see that the install-buildtools script should ideally
> still run on python34. The whole point is to make it easier to install the
> pre-built buildtools.
>

Python 3.5 is now required in bitbake/lib/bb/__init__.py
Which is needed for bb.utils.md5sum_file and sha256_file to check the
checksum of the download. I’d rather not have to rewrite those functions.

This means Debian-8 is a conundrum.


> Debian-8 is EOL in June 2020.
>
>
>> >  DEFAULT_BASE_URL: str = '
>> http://downloads.yoctoproject.org/releases/yocto'
>> >  DEFAULT_RELEASE: str = 'yocto-3.1_M2'
>> >  DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
>> > @@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
>> >
>> >
>> >  def main():
>> > +    global DEFAULT_INSTALL_DIR
>> >      global DEFAULT_BASE_URL
>> >      global DEFAULT_RELEASE
>> >      global DEFAULT_INSTALLER_VERSION
>> > @@ -73,6 +75,7 @@ def main():
>> >      filename: str = ""
>> >      release: str = ""
>> >      buildtools_url: str = ""
>> > +    install_dir: str = ""
>> >
>> >      parser = argparse.ArgumentParser(
>> >          description="Buildtools installation helper",
>> > @@ -87,6 +90,7 @@ def main():
>> >                               '(optional)\nRequires --url',
>> >                          action='store')
>> >      parser.add_argument('-d', '--directory',
>> > +                        default=DEFAULT_INSTALL_DIR,
>> >                          help='directory where buildtools SDK will be
>> installed (optional)',
>> >                          action='store')
>> >      parser.add_argument('-r', '--release',
>> > @@ -216,12 +220,12 @@ def main():
>> >          st = os.stat(tmpbuildtools)
>> >          os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
>> >          logger.debug(os.stat(tmpbuildtools))
>> > -        install_dir = "/opt/poky/%s" % args.installer_version
>> >          if args.directory:
>> >              install_dir = args.directory
>> >              ret = subprocess.call("%s -d %s -y" %
>> >                                    (tmpbuildtools, install_dir),
>> shell=True)
>> >          else:
>> > +            install_dir = "/opt/poky/%s" % args.installer_version
>> >              ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
>> >          if ret != 0:
>> >              logger.error("Could not run buildtools installer")
>> > @@ -238,7 +242,8 @@ def main():
>> >              tool = 'gcc'
>> >          else:
>> >              tool = 'tar'
>> > -        proc = subprocess.run("source
>> %s/environment-setup-x86_64-pokysdk-linux && which %s" %
>> > +        logger.debug("install_dir: %s" % install_dir)
>> > +        proc = subprocess.run(".
>> %s/environment-setup-x86_64-pokysdk-linux && which %s" %
>> >                                (install_dir, tool),
>> >                                shell=True, stdout=subprocess.PIPE)
>> >          which_tool = proc.stdout.decode("utf-8")
>> > --
>> > 2.24.0
>> >
>> >
>>
>> 
>

[-- Attachment #2: Type: text/html, Size: 9285 bytes --]

Re: [OE-core] [PATCH 1/4] scripts/install-buildtools: improvements

[Andre McCurdy]

On Wed, Apr 1, 2020 at 3:02 PM Tim Orling <ticotimo@gmail.com> wrote:
> On Wed, Apr 1, 2020 at 2:32 PM Tim Orling via lists.openembedded.org <ticotimo=gmail.com@lists.openembedded.org> wrote:
>> On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>>>
>>> On Mon, Mar 30, 2020 at 12:43 PM Tim Orling
>>> <timothy.t.orling@linux.intel.com> wrote:
>>> >
>>> > * Install directory defaults to scripts/../buildtools
>>> >   e.g. --directory is set by default
>>> >   This avoids the user having to type in their sudo password
>>> >   to install in /opt/poky/<installer-version>
>>> >
>>> > * Use "." rather than "source" for sourcing the environment script
>>> >   as not all distros (e.g. Debian) have "source" by default.
>>> >
>>> > * Add buildtools/ to .gitignore
>>> >
>>> > * Fix typos in example usage (--install-version -> --installer-version)
>>> >
>>> > [YOCTO #13832]
>>> >
>>> > Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
>>> > ---
>>> >  .gitignore                 |  1 +
>>> >  scripts/install-buildtools | 13 +++++++++----
>>> >  2 files changed, 10 insertions(+), 4 deletions(-)
>>> >
>>> > diff --git a/.gitignore b/.gitignore
>>> > index d0e6b2fb89..b66d371aac 100644
>>> > --- a/.gitignore
>>> > +++ b/.gitignore
>>> > @@ -9,6 +9,7 @@ pstage/
>>> >  scripts/oe-git-proxy-socks
>>> >  sources/
>>> >  meta-*/
>>> > +buildtools/
>>> >  !meta-skeleton
>>> >  !meta-selftest
>>> >  hob-image-*.bb
>>> > diff --git a/scripts/install-buildtools b/scripts/install-buildtools
>>> > index 0947e9c4d6..49cab1345a 100755
>>> > --- a/scripts/install-buildtools
>>> > +++ b/scripts/install-buildtools
>>> > @@ -17,7 +17,7 @@
>>> >  #        $ install-buildtools \
>>> >  #          --base-url http://downloads.yoctoproject.org/releases/yocto \
>>> >  #          --release yocto-3.1_M2 \
>>> > -#          --install-version 3.0+snapshot
>>> > +#          --installer-version 3.0+snapshot
>>> >  #          --build-date 202000122
>>> >  #
>>> >  #  Example usage (standard buildtools from release):
>>> > @@ -29,7 +29,7 @@
>>> >  #        $ install-buildtools --without-extended-buildtools \
>>> >  #          --base-url http://downloads.yoctoproject.org/releases/yocto \
>>> >  #          --release yocto-3.0.2 \
>>> > -#          --install-version 3.0.2
>>> > +#          --installer-version 3.0.2
>>> >  #
>>> >
>>> >  import argparse
>>> > @@ -59,6 +59,7 @@ if not bitbakepath:
>>> >  PROGNAME = 'install-buildtools'
>>> >  logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
>>> >
>>> > +DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools')
>>>
>>> It looks like there are some assumptions about the minimum version of
>>> python3 on the host in order to even be able to run
>>> install-buildtools. Should that be checked explicitly?
>>
>>
>> To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.
>>
>> The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.
>>
>> Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.
>
> Python 3.5 is now required in bitbake/lib/bb/__init__.py
> Which is needed for bb.utils.md5sum_file and sha256_file to check the checksum of the download. I’d rather not have to rewrite those functions.

Looks like copy and pasting ~30 lines? Maybe not even much of a net
increase in code size if you can remove the existing code to check
that bitbake libs are available...

def _hasher(method, filename):
    import mmap

    with open(filename, "rb") as f:
        try:
            with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
                for chunk in iter(lambda: mm.read(8192), b''):
                    method.update(chunk)
        except ValueError:
            # You can't mmap() an empty file so silence this exception
            pass
    return method.hexdigest()

def md5_file(filename):
    """
    Return the hex string representation of the MD5 checksum of filename.
    """
    import hashlib
    return _hasher(hashlib.md5(), filename)

def sha256_file(filename):
    """
    Return the hex string representation of the 256-bit SHA checksum of
    filename.
    """
    import hashlib
    return _hasher(hashlib.sha256(), filename)


>> Debian-8 is EOL in June 2020.
>>
>>>
>>> >  DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto'
>>> >  DEFAULT_RELEASE: str = 'yocto-3.1_M2'
>>> >  DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
>>> > @@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
>>> >
>>> >
>>> >  def main():
>>> > +    global DEFAULT_INSTALL_DIR
>>> >      global DEFAULT_BASE_URL
>>> >      global DEFAULT_RELEASE
>>> >      global DEFAULT_INSTALLER_VERSION
>>> > @@ -73,6 +75,7 @@ def main():
>>> >      filename: str = ""
>>> >      release: str = ""
>>> >      buildtools_url: str = ""
>>> > +    install_dir: str = ""
>>> >
>>> >      parser = argparse.ArgumentParser(
>>> >          description="Buildtools installation helper",
>>> > @@ -87,6 +90,7 @@ def main():
>>> >                               '(optional)\nRequires --url',
>>> >                          action='store')
>>> >      parser.add_argument('-d', '--directory',
>>> > +                        default=DEFAULT_INSTALL_DIR,
>>> >                          help='directory where buildtools SDK will be installed (optional)',
>>> >                          action='store')
>>> >      parser.add_argument('-r', '--release',
>>> > @@ -216,12 +220,12 @@ def main():
>>> >          st = os.stat(tmpbuildtools)
>>> >          os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
>>> >          logger.debug(os.stat(tmpbuildtools))
>>> > -        install_dir = "/opt/poky/%s" % args.installer_version
>>> >          if args.directory:
>>> >              install_dir = args.directory
>>> >              ret = subprocess.call("%s -d %s -y" %
>>> >                                    (tmpbuildtools, install_dir), shell=True)
>>> >          else:
>>> > +            install_dir = "/opt/poky/%s" % args.installer_version
>>> >              ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
>>> >          if ret != 0:
>>> >              logger.error("Could not run buildtools installer")
>>> > @@ -238,7 +242,8 @@ def main():
>>> >              tool = 'gcc'
>>> >          else:
>>> >              tool = 'tar'
>>> > -        proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" %
>>> > +        logger.debug("install_dir: %s" % install_dir)
>>> > +        proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" %
>>> >                                (install_dir, tool),
>>> >                                shell=True, stdout=subprocess.PIPE)
>>> >          which_tool = proc.stdout.decode("utf-8")
>>> > --
>>> > 2.24.0
>>> >
>>> >
>>>
>> 

Re: [OE-core] [PATCH 1/4] scripts/install-buildtools: improvements

[Richard Purdie]

On Wed, 2020-04-01 at 15:09 -0700, Andre McCurdy wrote:
> On Wed, Apr 1, 2020 at 3:02 PM Tim Orling <ticotimo@gmail.com> wrote:
> > On Wed, Apr 1, 2020 at 2:32 PM Tim Orling via
> > lists.openembedded.org <ticotimo=gmail.com@lists.openembedded.org>
> > wrote:
> > > On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <armccurdy@gmail.com
> > > > wrote:
> > > > It looks like there are some assumptions about the minimum
> > > > version of
> > > > python3 on the host in order to even be able to run
> > > > install-buildtools. Should that be checked explicitly?
> > > 
> > > To be honest, everything was written on CentOS-7, where ‘python3’
> > > is now python36.
> > > 
> > > The remaining supported holdout for 3.4 is Debian-8 (Jessie).
> > > Which also happens to have tar 1.27. At least it has git 2.1.4,
> > > so it’s not a complete mess.
> > > 
> > > Python 3.4 is already EOL. We should not support it for the
> > > complete build system, but I can see that the install-buildtools
> > > script should ideally still run on python34. The whole point is
> > > to make it easier to install the pre-built buildtools.
> > 
> > Python 3.5 is now required in bitbake/lib/bb/__init__.py
> > Which is needed for bb.utils.md5sum_file and sha256_file to check
> > the checksum of the download. I’d rather not have to rewrite those
> > functions.
> 
> Looks like copy and pasting ~30 lines? Maybe not even much of a net
> increase in code size if you can remove the existing code to check
> that bitbake libs are available...
> 
> def _hasher(method, filename):
>     import mmap
> 
>     with open(filename, "rb") as f:
>         try:
>             with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as
> mm:
>                 for chunk in iter(lambda: mm.read(8192), b''):
>                     method.update(chunk)
>         except ValueError:
>             # You can't mmap() an empty file so silence this
> exception
>             pass
>     return method.hexdigest()
> 
> def md5_file(filename):
>     """
>     Return the hex string representation of the MD5 checksum of
> filename.
>     """
>     import hashlib
>     return _hasher(hashlib.md5(), filename)
> 
> def sha256_file(filename):
>     """
>     Return the hex string representation of the 256-bit SHA checksum
> of
>     filename.
>     """
>     import hashlib
>     return _hasher(hashlib.sha256(), filename)

Can you send a patch against master-next please? The buildtools
installer is one place we probably do want to support older pythons and
it shouldn't be too onnerous.

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] scripts/install-buildtools: improvements

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 3130 bytes --]



> On Apr 1, 2020, at 3:16 PM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> 
> On Wed, 2020-04-01 at 15:09 -0700, Andre McCurdy wrote:
>> On Wed, Apr 1, 2020 at 3:02 PM Tim Orling <ticotimo@gmail.com> wrote:
>>> On Wed, Apr 1, 2020 at 2:32 PM Tim Orling via
>>> lists.openembedded.org <ticotimo=gmail.com@lists.openembedded.org>
>>> wrote:
>>>> On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <armccurdy@gmail.com
>>>>> wrote:
>>>>> It looks like there are some assumptions about the minimum
>>>>> version of
>>>>> python3 on the host in order to even be able to run
>>>>> install-buildtools. Should that be checked explicitly?
>>>> 
>>>> To be honest, everything was written on CentOS-7, where ‘python3’
>>>> is now python36.
>>>> 
>>>> The remaining supported holdout for 3.4 is Debian-8 (Jessie).
>>>> Which also happens to have tar 1.27. At least it has git 2.1.4,
>>>> so it’s not a complete mess.
>>>> 
>>>> Python 3.4 is already EOL. We should not support it for the
>>>> complete build system, but I can see that the install-buildtools
>>>> script should ideally still run on python34. The whole point is
>>>> to make it easier to install the pre-built buildtools.
>>> 
>>> Python 3.5 is now required in bitbake/lib/bb/__init__.py
>>> Which is needed for bb.utils.md5sum_file and sha256_file to check
>>> the checksum of the download. I’d rather not have to rewrite those
>>> functions.
>> 
>> Looks like copy and pasting ~30 lines? Maybe not even much of a net
>> increase in code size if you can remove the existing code to check
>> that bitbake libs are available...
>> 
>> def _hasher(method, filename):
>>    import mmap
>> 
>>    with open(filename, "rb") as f:
>>        try:
>>            with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as
>> mm:
>>                for chunk in iter(lambda: mm.read(8192), b''):
>>                    method.update(chunk)
>>        except ValueError:
>>            # You can't mmap() an empty file so silence this
>> exception
>>            pass
>>    return method.hexdigest()
>> 
>> def md5_file(filename):
>>    """
>>    Return the hex string representation of the MD5 checksum of
>> filename.
>>    """
>>    import hashlib
>>    return _hasher(hashlib.md5(), filename)
>> 
>> def sha256_file(filename):
>>    """
>>    Return the hex string representation of the 256-bit SHA checksum
>> of
>>    filename.
>>    """
>>    import hashlib
>>    return _hasher(hashlib.sha256(), filename)
> 
> Can you send a patch against master-next please? The buildtools
> installer is one place we probably do want to support older pythons and
> it shouldn't be too onnerous.
> 

Submitted. The check for whether the install worked or not was the hardest part.
After many iterations, I just could not get subprocess.Popen to work as expected on python 3.4.
https://patchwork.openembedded.org/patch/171593/ <https://patchwork.openembedded.org/patch/171593/>

Thank you Andre for the testing and input. It takes a village to make a good release.

> Cheers,
> 
> Richard
> 
> 


[-- Attachment #2: Type: text/html, Size: 5397 bytes --]