[PATCH 0/7] Sanity class clean ups

[PATCH 0/7] Sanity class clean ups

[Joshua Lock]

My recent change to the sanity class to warn users when they don't have R/W
permission to their SSTATE_CACHE directory has caused at least a couple of users
trouble, including the Yocto project autobuilder.

I have been unable to reproduce the issue but after discussionwith Elizabeth and
Chris on #yocto I came up with the following series.

The series includes a revert my original change as it seems the simple logic has
severla edge cases. I replace it later in the series with a simple piggy-back
on the existing check_create_long_filename() call, where I make the suggestion
of using SSTATE_MIRRORS if that call fails with "Permission denied" when called
against SSTATE_DIR. This check has been in use for some time and, to the best of
my knowledge, doesn't trigger invalid failures.

Thanks to Chris and Elizabeth for pointers as to why this was failing.

Cheers,

Joshua

The following changes since commit e6333825c3482a559a0c0499e17f8f48d3042ddf:

  tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)

are available in the git repository at:
  git://git.openembedded.org/openembedded-core-contrib josh/sanity
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanity

Joshua Lock (7):
  Revert "sanity.bbclass: check user can read and write to SSTATE_DIR"
  sanity.bbclass: copy the data store and finalise before running
    checks
  sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp)
  sanity.bbclass: add newline to check_create_long_filename failure
    message
  sanity.bbclass: add extra information when SSTATE_CACHE unusable
  sanity.bbclass: catch an extra exception in
    check_create_long_filename
  sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed

 meta/classes/sanity.bbclass |  153 ++++++++++++++++++++++---------------------
 1 files changed, 78 insertions(+), 75 deletions(-)

-- 
1.7.7.6

[PATCH 1/7] Revert "sanity.bbclass: check user can read and write to SSTATE_DIR"

[Joshua Lock]

This has caused problems for several users, including the Yocto Project
autobuilder. Since the message was added in order to be more user friendly
revert the change.

This reverts commit 0c0c4efbf92bcf0f8942f17c18525a4b4ed1798c.
---
 meta/classes/sanity.bbclass |   17 +++--------------
 1 files changed, 3 insertions(+), 14 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index ff3c413..05545f4 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -96,16 +96,10 @@ def check_conf_exists(fn, data):
 def check_sanity_sstate_dir_change(sstate_dir, data):
     # Sanity checks to be done when the value of SSTATE_DIR changes
 
+    # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS)
     testmsg = ""
     if sstate_dir != "":
-        # Check that the user can read and write to SSTATE_DIR
-        sstatemsg = check_can_read_write_directory(sstate_dir) or None
-        if sstatemsg:
-            sstatemsg = sstatemsg + ". You could try using it as an SSTATE_MIRRORS instead of SSTATE_CACHE.\n"
-            testmsg = testmsg + sstatemsg
-        # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS)
-        testmsg = testmsg + check_create_long_filename(sstate_dir, "SSTATE_DIR")
-
+        testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR")
     return testmsg
 
 def check_sanity_tmpdir_change(tmpdir, data):
@@ -156,12 +150,7 @@ def check_create_long_filename(filepath, pathname):
         if errno == 36: # ENAMETOOLONG
             return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname
         else:
-            return "Failed to create a file in %s: %s\n" % (pathname, strerror)
-    return ""
-
-def check_can_read_write_directory(directory):
-    if not os.access(directory, os.R_OK|os.W_OK):
-       return "Insufficient permissions for %s" % directory
+            return "Failed to create a file in %s: %s" % (pathname, strerror)
     return ""
 
 def check_connectivity(d):
-- 
1.7.7.6

[PATCH 2/7] sanity.bbclass: copy the data store and finalise before running checks

[Joshua Lock]

At the ConfigParsed event the datastore has yet to be finalised and thus
appends and overrides have not been set.
To ensure the sanity check is being run against the configuration values
the user has set call finalize() on a copy of the datastore and pass that
for all sanity checks.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |  129 +++++++++++++++++++++++--------------------
 1 files changed, 69 insertions(+), 60 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 05545f4..5361f1f 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -182,8 +182,8 @@ def check_connectivity(d):
 
     return retval
 
-def check_supported_distro(e):
-    tested_distros = e.data.getVar('SANITY_TESTED_DISTROS', True)
+def check_supported_distro(sanity_data):
+    tested_distros = sanity_data.getVar('SANITY_TESTED_DISTROS', True)
     if not tested_distros:
         return
 
@@ -230,26 +230,26 @@ def check_supported_distro(e):
         bb.warn('Host distribution could not be determined; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.')
 
 # Checks we should only make if MACHINE is set correctly
-def check_sanity_validmachine(e):
+def check_sanity_validmachine(sanity_data):
     from bb import data
 
     messages = ""
 
     # Check TUNE_ARCH is set
-    if data.getVar('TUNE_ARCH', e.data, True) == 'INVALID':
+    if data.getVar('TUNE_ARCH', sanity_data, True) == 'INVALID':
         messages = messages + 'TUNE_ARCH is unset. Please ensure your MACHINE configuration includes a valid tune configuration file which will set this correctly.\n'
 
     # Check TARGET_ARCH is set correctly
-    if data.getVar('TARGE_ARCH', e.data, False) == '${TUNE_ARCH}':
+    if data.getVar('TARGE_ARCH', sanity_data, False) == '${TUNE_ARCH}':
         messages = messages + 'TARGET_ARCH is being overwritten, likely by your MACHINE configuration files.\nPlease use a valid tune configuration file which should set this correctly automatically\nand avoid setting this in the machine configuration. See the OE-Core mailing list for more information.\n'
     
     # Check TARGET_OS is set
-    if data.getVar('TARGET_OS', e.data, True) == 'INVALID':
+    if data.getVar('TARGET_OS', sanity_data, True) == 'INVALID':
         messages = messages + 'Please set TARGET_OS directly, or choose a MACHINE or DISTRO that does so.\n'
 
     # Check that we don't have duplicate entries in PACKAGE_ARCHS & that TUNE_PKGARCH is in PACKAGE_ARCHS
-    pkgarchs = data.getVar('PACKAGE_ARCHS', e.data, True)
-    tunepkg = data.getVar('TUNE_PKGARCH', e.data, True)
+    pkgarchs = data.getVar('PACKAGE_ARCHS', sanity_data, True)
+    tunepkg = data.getVar('TUNE_PKGARCH', sanity_data, True)
     tunefound = False
     seen = {}
     dups = []
@@ -271,7 +271,7 @@ def check_sanity_validmachine(e):
     return messages
 
 
-def check_sanity(e):
+def check_sanity(sanity_data):
     from bb import note, error, data, __version__
 
     try:
@@ -281,7 +281,7 @@ def check_sanity(e):
     import commands
 
     # Check the bitbake version meets minimum requirements
-    minversion = data.getVar('BB_MIN_VERSION', e.data , True)
+    minversion = data.getVar('BB_MIN_VERSION', sanity_data , True)
     if not minversion:
         # Hack: BB_MIN_VERSION hasn't been parsed yet so return 
         # and wait for the next call
@@ -303,42 +303,42 @@ def check_sanity(e):
         messages = messages + 'Bitbake version %s is required and version %s was found\n' % (minversion, __version__)
 
     # Check that the MACHINE is valid, if it is set
-    if data.getVar('MACHINE', e.data, True):
-        if not check_conf_exists("conf/machine/${MACHINE}.conf", e.data):
+    if data.getVar('MACHINE', sanity_data, True):
+        if not check_conf_exists("conf/machine/${MACHINE}.conf", sanity_data):
             messages = messages + 'Please set a valid MACHINE in your local.conf or environment\n'
         else:
-            messages = messages + check_sanity_validmachine(e)
+            messages = messages + check_sanity_validmachine(sanity_data)
     else:
         messages = messages + 'Please set a MACHINE in your local.conf or environment\n'
 
     # Check we are using a valid lacal.conf
-    current_conf  = data.getVar('CONF_VERSION', e.data, True)
-    conf_version =  data.getVar('LOCALCONF_VERSION', e.data, True)
+    current_conf  = data.getVar('CONF_VERSION', sanity_data, True)
+    conf_version =  data.getVar('LOCALCONF_VERSION', sanity_data, True)
 
     if current_conf != conf_version:
         messages = messages + "Your version of local.conf was generated from an older version of local.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/local.conf conf/local.conf.sample\" is a good way to visualise the changes.\n"
 
     # Check bblayers.conf is valid
-    current_lconf = data.getVar('LCONF_VERSION', e.data, True)
-    lconf_version = data.getVar('LAYER_CONF_VERSION', e.data, True)
+    current_lconf = data.getVar('LCONF_VERSION', sanity_data, True)
+    lconf_version = data.getVar('LAYER_CONF_VERSION', sanity_data, True)
     if current_lconf != lconf_version:
         messages = messages + "Your version of bblayers.conf was generated from an older version of bblayers.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/bblayers.conf conf/bblayers.conf.sample\" is a good way to visualise the changes.\n"
 
     # If we have a site.conf, check it's valid
-    if check_conf_exists("conf/site.conf", e.data):
-        current_sconf = data.getVar('SCONF_VERSION', e.data, True)
-        sconf_version = data.getVar('SITE_CONF_VERSION', e.data, True)
+    if check_conf_exists("conf/site.conf", sanity_data):
+        current_sconf = data.getVar('SCONF_VERSION', sanity_data, True)
+        sconf_version = data.getVar('SITE_CONF_VERSION', sanity_data, True)
         if current_sconf != sconf_version:
             messages = messages + "Your version of site.conf was generated from an older version of site.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/site.conf conf/site.conf.sample\" is a good way to visualise the changes.\n"
 
-    assume_provided = data.getVar('ASSUME_PROVIDED', e.data , True).split()
+    assume_provided = data.getVar('ASSUME_PROVIDED', sanity_data , True).split()
     # Check user doesn't have ASSUME_PROVIDED = instead of += in local.conf
     if "diffstat-native" not in assume_provided:
         messages = messages + 'Please use ASSUME_PROVIDED +=, not ASSUME_PROVIDED = in your local.conf\n'
 
     # Check that DL_DIR is set, exists and is writable. In theory, we should never even hit the check if DL_DIR isn't 
     # set, since so much relies on it being set.
-    dldir = data.getVar('DL_DIR', e.data, True)
+    dldir = data.getVar('DL_DIR', sanity_data, True)
     if not dldir:
         messages = messages + "DL_DIR is not set. Your environment is misconfigured, check that DL_DIR is set, and if the directory exists, that it is writable. \n"
     if os.path.exists(dldir) and not os.access(dldir, os.W_OK):
@@ -346,33 +346,33 @@ def check_sanity(e):
     
     # Check that the DISTRO is valid, if set
     # need to take into account DISTRO renaming DISTRO
-    distro = data.getVar('DISTRO', e.data, True)
+    distro = data.getVar('DISTRO', sanity_data, True)
     if distro:
-        if not ( check_conf_exists("conf/distro/${DISTRO}.conf", e.data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", e.data) ):
-            messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", e.data, True )
+        if not ( check_conf_exists("conf/distro/${DISTRO}.conf", sanity_data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", sanity_data) ):
+            messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", sanity_data, True )
 
     missing = ""
 
-    if not check_app_exists("${MAKE}", e.data):
+    if not check_app_exists("${MAKE}", sanity_data):
         missing = missing + "GNU make,"
 
-    if not check_app_exists('${BUILD_PREFIX}gcc', e.data):
-        missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", e.data, True)
+    if not check_app_exists('${BUILD_PREFIX}gcc', sanity_data):
+        missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", sanity_data, True)
 
-    if not check_app_exists('${BUILD_PREFIX}g++', e.data):
-        missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", e.data, True)
+    if not check_app_exists('${BUILD_PREFIX}g++', sanity_data):
+        missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", sanity_data, True)
 
-    required_utilities = e.data.getVar('SANITY_REQUIRED_UTILITIES', True)
+    required_utilities = sanity_data.getVar('SANITY_REQUIRED_UTILITIES', True)
 
     if "qemu-native" in assume_provided:
-        if not check_app_exists("qemu-arm", e.data):
+        if not check_app_exists("qemu-arm", sanity_data):
             messages = messages + "qemu-native was in ASSUME_PROVIDED but the QEMU binaries (qemu-arm) can't be found in PATH"
 
-    paths = data.getVar('PATH', e.data, True).split(":")
+    paths = data.getVar('PATH', sanity_data, True).split(":")
     if "." in paths or "" in paths:
         messages = messages + "PATH contains '.' or '', which will break the build, please remove this."
 
-    if data.getVar('TARGET_ARCH', e.data, True) == "arm":
+    if data.getVar('TARGET_ARCH', sanity_data, True) == "arm":
         # This path is no longer user-readable in modern (very recent) Linux
         try:
             if os.path.exists("/proc/sys/vm/mmap_min_addr"):
@@ -386,7 +386,7 @@ def check_sanity(e):
             pass
 
     for util in required_utilities.split():
-        if not check_app_exists( util, e.data ):
+        if not check_app_exists( util, sanity_data ):
             missing = missing + "%s," % util
 
     if missing != "":
@@ -397,13 +397,13 @@ def check_sanity(e):
     if pseudo_msg != "":
         messages = messages + pseudo_msg + '\n'
 
-    check_supported_distro(e)
-    toolchain_msg = check_toolchain(e.data)
+    check_supported_distro(sanity_data)
+    toolchain_msg = check_toolchain(sanity_data)
     if toolchain_msg != "":
         messages = messages + toolchain_msg + '\n'
 
     # Check if DISPLAY is set if IMAGETEST is set
-    if not data.getVar( 'DISPLAY', e.data, True ) and data.getVar( 'IMAGETEST', e.data, True ) == 'qemu':
+    if not data.getVar( 'DISPLAY', sanity_data, True ) and data.getVar( 'IMAGETEST', sanity_data, True ) == 'qemu':
         messages = messages + 'qemuimagetest needs a X desktop to start qemu, please set DISPLAY correctly (e.g. DISPLAY=:1.0)\n'
 
     omask = os.umask(022)
@@ -411,11 +411,11 @@ def check_sanity(e):
         messages = messages + "Please use a umask which allows a+rx and u+rwx\n"
     os.umask(omask)
 
-    oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', e.data, True )
+    oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', sanity_data, True )
     if not oes_bb_conf:
         messages = messages + 'You do not include OpenEmbeddeds version of conf/bitbake.conf. This means your environment is misconfigured, in particular check BBPATH.\n'
 
-    nolibs = data.getVar('NO32LIBS', e.data, True)
+    nolibs = data.getVar('NO32LIBS', sanity_data, True)
     if not nolibs:
         lib32path = '/lib'
         if os.path.exists('/lib64') and ( os.path.islink('/lib64') or os.path.islink('/lib') ):
@@ -424,8 +424,8 @@ def check_sanity(e):
         if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'):
             messages = messages + "You have a 32-bit libc, but no 32-bit headers.  You must install the 32-bit libc headers.\n"
 
-    tmpdir = data.getVar('TMPDIR', e.data, True)
-    sstate_dir = data.getVar('SSTATE_DIR', e.data, True)
+    tmpdir = data.getVar('TMPDIR', sanity_data, True)
+    sstate_dir = data.getVar('SSTATE_DIR', sanity_data, True)
 
     # Check saved sanity info
     last_sanity_version = 0
@@ -442,16 +442,16 @@ def check_sanity(e):
             if line.startswith('SSTATE_DIR'):
                 last_sstate_dir = line.split()[1]
     
-    sanity_version = int(data.getVar('SANITY_VERSION', e.data, True) or 1)
+    sanity_version = int(data.getVar('SANITY_VERSION', sanity_data, True) or 1)
     if last_sanity_version < sanity_version: 
-        messages = messages + check_sanity_version_change(e.data)
-        messages = messages + check_sanity_tmpdir_change(tmpdir, e.data)
-        messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data)
+        messages = messages + check_sanity_version_change(sanity_data)
+        messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data)
+        messages = messages + check_sanity_sstate_dir_change(sstate_dir, sanity_data)
     else: 
         if last_tmpdir != tmpdir:
-            messages = messages + check_sanity_tmpdir_change(tmpdir, e.data)
+            messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data)
         if last_sstate_dir != sstate_dir:
-            messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data)
+            messages = messages + check_sanity_sstate_dir_change(sstate_dir, sanity_data)
 
     if os.path.exists("conf") and not messages:
         f = file(sanityverfile, 'w')
@@ -476,8 +476,8 @@ def check_sanity(e):
     #
     # Check the 'ABI' of TMPDIR
     #
-    current_abi = data.getVar('OELAYOUT_ABI', e.data, True)
-    abifile = data.getVar('SANITY_ABIFILE', e.data, True)
+    current_abi = data.getVar('OELAYOUT_ABI', sanity_data, True)
+    abifile = data.getVar('SANITY_ABIFILE', sanity_data, True)
     if os.path.exists(abifile):
         f = file(abifile, "r")
         abi = f.read().strip()
@@ -486,16 +486,16 @@ def check_sanity(e):
             f.write(current_abi)
         elif abi == "2" and current_abi == "3":
             bb.note("Converting staging from layout version 2 to layout version 3")
-            os.system(e.data.expand("mv ${TMPDIR}/staging ${TMPDIR}/sysroots"))
-            os.system(e.data.expand("ln -s sysroots ${TMPDIR}/staging"))
-            os.system(e.data.expand("cd ${TMPDIR}/stamps; for i in */*do_populate_staging; do new=`echo $i | sed -e 's/do_populate_staging/do_populate_sysroot/'`; mv $i $new; done"))
+            os.system(sanity_data.expand("mv ${TMPDIR}/staging ${TMPDIR}/sysroots"))
+            os.system(sanity_data.expand("ln -s sysroots ${TMPDIR}/staging"))
+            os.system(sanity_data.expand("cd ${TMPDIR}/stamps; for i in */*do_populate_staging; do new=`echo $i | sed -e 's/do_populate_staging/do_populate_sysroot/'`; mv $i $new; done"))
             f = file(abifile, "w")
             f.write(current_abi)
         elif abi == "3" and current_abi == "4":
             bb.note("Converting staging layout from version 3 to layout version 4")
-            if os.path.exists(e.data.expand("${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}")):
-                os.system(e.data.expand("mv ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS} ${STAGING_BINDIR_CROSS}"))
-                os.system(e.data.expand("ln -s ${STAGING_BINDIR_CROSS} ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}"))
+            if os.path.exists(sanity_data.expand("${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}")):
+                os.system(sanity_data.expand("mv ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS} ${STAGING_BINDIR_CROSS}"))
+                os.system(sanity_data.expand("ln -s ${STAGING_BINDIR_CROSS} ${STAGING_DIR_NATIVE}${bindir_native}/${MULTIMACH_HOST_SYS}"))
 
             f = file(abifile, "w")
             f.write(current_abi)
@@ -503,7 +503,7 @@ def check_sanity(e):
             messages = messages + "Staging layout has changed. The cross directory has been deprecated and cross packages are now built under the native sysroot.\nThis requires a rebuild.\n"
         elif abi == "5" and current_abi == "6":
             bb.note("Converting staging layout from version 5 to layout version 6")
-            os.system(e.data.expand("mv ${TMPDIR}/pstagelogs ${SSTATE_MANIFESTS}"))
+            os.system(sanity_data.expand("mv ${TMPDIR}/pstagelogs ${SSTATE_MANIFESTS}"))
             f = file(abifile, "w")
             f.write(current_abi)
         elif abi == "7" and current_abi == "8":
@@ -516,7 +516,7 @@ def check_sanity(e):
         f.write(current_abi)
     f.close()
 
-    oeroot = data.getVar('COREBASE', e.data)
+    oeroot = data.getVar('COREBASE', sanity_data)
     if oeroot.find ('+') != -1:
         messages = messages + "Error, you have an invalid character (+) in your COREBASE directory path. Please move the installation to a directory which doesn't include a +."
     elif oeroot.find (' ') != -1:
@@ -525,12 +525,21 @@ def check_sanity(e):
     if messages != "":
         raise_sanity_error(messages)
 
+# Create a copy of the datastore and finalise it to ensure appends and 
+# overrides are set - the datastore has yet to be finalised at ConfigParsed
+def copy_data(e):
+    sanity_data = bb.data.createCopy(e.data)
+    sanity_data.finalize()
+    return sanity_data
+
 addhandler check_sanity_eventhandler
 python check_sanity_eventhandler() {
     if bb.event.getName(e) == "ConfigParsed" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1":
-        check_sanity(e)
+        sanity_data = copy_data(e)
+        check_sanity(sanity_data)
     elif bb.event.getName(e) == "SanityCheck":
-        check_sanity(e)
+        sanity_data = copy_data(e)
+        check_sanity(sanity_data)
         bb.event.fire(bb.event.SanityCheckPassed(), e.data)
 
     return
-- 
1.7.7.6

[PATCH 3/7] sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp)

[Joshua Lock]

Replace calls to data.getVar(VARIABLE, data_object, expand) to
direct calls to the getVar method the the data_object.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |   60 +++++++++++++++++++++---------------------
 1 files changed, 30 insertions(+), 30 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 5361f1f..5cf9ea1 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -236,20 +236,20 @@ def check_sanity_validmachine(sanity_data):
     messages = ""
 
     # Check TUNE_ARCH is set
-    if data.getVar('TUNE_ARCH', sanity_data, True) == 'INVALID':
+    if sanity_data.getVar('TUNE_ARCH', True) == 'INVALID':
         messages = messages + 'TUNE_ARCH is unset. Please ensure your MACHINE configuration includes a valid tune configuration file which will set this correctly.\n'
 
     # Check TARGET_ARCH is set correctly
-    if data.getVar('TARGE_ARCH', sanity_data, False) == '${TUNE_ARCH}':
+    if sanity_data.getVar('TARGE_ARCH', False) == '${TUNE_ARCH}':
         messages = messages + 'TARGET_ARCH is being overwritten, likely by your MACHINE configuration files.\nPlease use a valid tune configuration file which should set this correctly automatically\nand avoid setting this in the machine configuration. See the OE-Core mailing list for more information.\n'
     
     # Check TARGET_OS is set
-    if data.getVar('TARGET_OS', sanity_data, True) == 'INVALID':
+    if sanity_data.getVar('TARGET_OS', True) == 'INVALID':
         messages = messages + 'Please set TARGET_OS directly, or choose a MACHINE or DISTRO that does so.\n'
 
     # Check that we don't have duplicate entries in PACKAGE_ARCHS & that TUNE_PKGARCH is in PACKAGE_ARCHS
-    pkgarchs = data.getVar('PACKAGE_ARCHS', sanity_data, True)
-    tunepkg = data.getVar('TUNE_PKGARCH', sanity_data, True)
+    pkgarchs = sanity_data.getVar('PACKAGE_ARCHS', True)
+    tunepkg = sanity_data.getVar('TUNE_PKGARCH', True)
     tunefound = False
     seen = {}
     dups = []
@@ -281,7 +281,7 @@ def check_sanity(sanity_data):
     import commands
 
     # Check the bitbake version meets minimum requirements
-    minversion = data.getVar('BB_MIN_VERSION', sanity_data , True)
+    minversion = sanity_data.getVar('BB_MIN_VERSION', True)
     if not minversion:
         # Hack: BB_MIN_VERSION hasn't been parsed yet so return 
         # and wait for the next call
@@ -303,7 +303,7 @@ def check_sanity(sanity_data):
         messages = messages + 'Bitbake version %s is required and version %s was found\n' % (minversion, __version__)
 
     # Check that the MACHINE is valid, if it is set
-    if data.getVar('MACHINE', sanity_data, True):
+    if sanity_data.getVar('MACHINE', True):
         if not check_conf_exists("conf/machine/${MACHINE}.conf", sanity_data):
             messages = messages + 'Please set a valid MACHINE in your local.conf or environment\n'
         else:
@@ -312,33 +312,33 @@ def check_sanity(sanity_data):
         messages = messages + 'Please set a MACHINE in your local.conf or environment\n'
 
     # Check we are using a valid lacal.conf
-    current_conf  = data.getVar('CONF_VERSION', sanity_data, True)
-    conf_version =  data.getVar('LOCALCONF_VERSION', sanity_data, True)
+    current_conf  = sanity_data.getVar('CONF_VERSION', True)
+    conf_version =  sanity_data.getVar('LOCALCONF_VERSION', True)
 
     if current_conf != conf_version:
         messages = messages + "Your version of local.conf was generated from an older version of local.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/local.conf conf/local.conf.sample\" is a good way to visualise the changes.\n"
 
     # Check bblayers.conf is valid
-    current_lconf = data.getVar('LCONF_VERSION', sanity_data, True)
-    lconf_version = data.getVar('LAYER_CONF_VERSION', sanity_data, True)
+    current_lconf = sanity_data.getVar('LCONF_VERSION', True)
+    lconf_version = sanity_data.getVar('LAYER_CONF_VERSION', True)
     if current_lconf != lconf_version:
         messages = messages + "Your version of bblayers.conf was generated from an older version of bblayers.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/bblayers.conf conf/bblayers.conf.sample\" is a good way to visualise the changes.\n"
 
     # If we have a site.conf, check it's valid
     if check_conf_exists("conf/site.conf", sanity_data):
-        current_sconf = data.getVar('SCONF_VERSION', sanity_data, True)
-        sconf_version = data.getVar('SITE_CONF_VERSION', sanity_data, True)
+        current_sconf = sanity_data.getVar('SCONF_VERSION', True)
+        sconf_version = sanity_data.getVar('SITE_CONF_VERSION', True)
         if current_sconf != sconf_version:
             messages = messages + "Your version of site.conf was generated from an older version of site.conf.sample and there have been updates made to this file. Please compare the two files and merge any changes before continuing.\nMatching the version numbers will remove this message.\n\"meld conf/site.conf conf/site.conf.sample\" is a good way to visualise the changes.\n"
 
-    assume_provided = data.getVar('ASSUME_PROVIDED', sanity_data , True).split()
+    assume_provided = sanity_data.getVar('ASSUME_PROVIDED', True).split()
     # Check user doesn't have ASSUME_PROVIDED = instead of += in local.conf
     if "diffstat-native" not in assume_provided:
         messages = messages + 'Please use ASSUME_PROVIDED +=, not ASSUME_PROVIDED = in your local.conf\n'
 
     # Check that DL_DIR is set, exists and is writable. In theory, we should never even hit the check if DL_DIR isn't 
     # set, since so much relies on it being set.
-    dldir = data.getVar('DL_DIR', sanity_data, True)
+    dldir = sanity_data.getVar('DL_DIR', True)
     if not dldir:
         messages = messages + "DL_DIR is not set. Your environment is misconfigured, check that DL_DIR is set, and if the directory exists, that it is writable. \n"
     if os.path.exists(dldir) and not os.access(dldir, os.W_OK):
@@ -346,10 +346,10 @@ def check_sanity(sanity_data):
     
     # Check that the DISTRO is valid, if set
     # need to take into account DISTRO renaming DISTRO
-    distro = data.getVar('DISTRO', sanity_data, True)
+    distro = sanity_data.getVar('DISTRO', True)
     if distro:
         if not ( check_conf_exists("conf/distro/${DISTRO}.conf", sanity_data) or check_conf_exists("conf/distro/include/${DISTRO}.inc", sanity_data) ):
-            messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % data.getVar("DISTRO", sanity_data, True )
+            messages = messages + "DISTRO '%s' not found. Please set a valid DISTRO in your local.conf\n" % sanity_data.getVar("DISTRO", True )
 
     missing = ""
 
@@ -357,10 +357,10 @@ def check_sanity(sanity_data):
         missing = missing + "GNU make,"
 
     if not check_app_exists('${BUILD_PREFIX}gcc', sanity_data):
-        missing = missing + "C Compiler (%sgcc)," % data.getVar("BUILD_PREFIX", sanity_data, True)
+        missing = missing + "C Compiler (%sgcc)," % sanity_data.getVar("BUILD_PREFIX", True)
 
     if not check_app_exists('${BUILD_PREFIX}g++', sanity_data):
-        missing = missing + "C++ Compiler (%sg++)," % data.getVar("BUILD_PREFIX", sanity_data, True)
+        missing = missing + "C++ Compiler (%sg++)," % sanity_data.getVar("BUILD_PREFIX", True)
 
     required_utilities = sanity_data.getVar('SANITY_REQUIRED_UTILITIES', True)
 
@@ -368,11 +368,11 @@ def check_sanity(sanity_data):
         if not check_app_exists("qemu-arm", sanity_data):
             messages = messages + "qemu-native was in ASSUME_PROVIDED but the QEMU binaries (qemu-arm) can't be found in PATH"
 
-    paths = data.getVar('PATH', sanity_data, True).split(":")
+    paths = sanity_data.getVar('PATH', True).split(":")
     if "." in paths or "" in paths:
         messages = messages + "PATH contains '.' or '', which will break the build, please remove this."
 
-    if data.getVar('TARGET_ARCH', sanity_data, True) == "arm":
+    if sanity_data.getVar('TARGET_ARCH', True) == "arm":
         # This path is no longer user-readable in modern (very recent) Linux
         try:
             if os.path.exists("/proc/sys/vm/mmap_min_addr"):
@@ -403,7 +403,7 @@ def check_sanity(sanity_data):
         messages = messages + toolchain_msg + '\n'
 
     # Check if DISPLAY is set if IMAGETEST is set
-    if not data.getVar( 'DISPLAY', sanity_data, True ) and data.getVar( 'IMAGETEST', sanity_data, True ) == 'qemu':
+    if not sanity_data.getVar( 'DISPLAY', True ) and sanity_data.getVar( 'IMAGETEST', True ) == 'qemu':
         messages = messages + 'qemuimagetest needs a X desktop to start qemu, please set DISPLAY correctly (e.g. DISPLAY=:1.0)\n'
 
     omask = os.umask(022)
@@ -411,11 +411,11 @@ def check_sanity(sanity_data):
         messages = messages + "Please use a umask which allows a+rx and u+rwx\n"
     os.umask(omask)
 
-    oes_bb_conf = data.getVar( 'OES_BITBAKE_CONF', sanity_data, True )
+    oes_bb_conf = sanity_data.getVar( 'OES_BITBAKE_CONF', True)
     if not oes_bb_conf:
         messages = messages + 'You do not include OpenEmbeddeds version of conf/bitbake.conf. This means your environment is misconfigured, in particular check BBPATH.\n'
 
-    nolibs = data.getVar('NO32LIBS', sanity_data, True)
+    nolibs = sanity_data.getVar('NO32LIBS', True)
     if not nolibs:
         lib32path = '/lib'
         if os.path.exists('/lib64') and ( os.path.islink('/lib64') or os.path.islink('/lib') ):
@@ -424,8 +424,8 @@ def check_sanity(sanity_data):
         if os.path.exists('%s/libc.so.6' % lib32path) and not os.path.exists('/usr/include/gnu/stubs-32.h'):
             messages = messages + "You have a 32-bit libc, but no 32-bit headers.  You must install the 32-bit libc headers.\n"
 
-    tmpdir = data.getVar('TMPDIR', sanity_data, True)
-    sstate_dir = data.getVar('SSTATE_DIR', sanity_data, True)
+    tmpdir = sanity_data.getVar('TMPDIR', True)
+    sstate_dir = sanity_data.getVar('SSTATE_DIR', True)
 
     # Check saved sanity info
     last_sanity_version = 0
@@ -442,7 +442,7 @@ def check_sanity(sanity_data):
             if line.startswith('SSTATE_DIR'):
                 last_sstate_dir = line.split()[1]
     
-    sanity_version = int(data.getVar('SANITY_VERSION', sanity_data, True) or 1)
+    sanity_version = int(sanity_data.getVar('SANITY_VERSION', True) or 1)
     if last_sanity_version < sanity_version: 
         messages = messages + check_sanity_version_change(sanity_data)
         messages = messages + check_sanity_tmpdir_change(tmpdir, sanity_data)
@@ -476,8 +476,8 @@ def check_sanity(sanity_data):
     #
     # Check the 'ABI' of TMPDIR
     #
-    current_abi = data.getVar('OELAYOUT_ABI', sanity_data, True)
-    abifile = data.getVar('SANITY_ABIFILE', sanity_data, True)
+    current_abi = sanity_data.getVar('OELAYOUT_ABI', True)
+    abifile = sanity_data.getVar('SANITY_ABIFILE', True)
     if os.path.exists(abifile):
         f = file(abifile, "r")
         abi = f.read().strip()
@@ -516,7 +516,7 @@ def check_sanity(sanity_data):
         f.write(current_abi)
     f.close()
 
-    oeroot = data.getVar('COREBASE', sanity_data)
+    oeroot = sanity_data.getVar('COREBASE')
     if oeroot.find ('+') != -1:
         messages = messages + "Error, you have an invalid character (+) in your COREBASE directory path. Please move the installation to a directory which doesn't include a +."
     elif oeroot.find (' ') != -1:
-- 
1.7.7.6

[PATCH 4/7] sanity.bbclass: add newline to check_create_long_filename failure message

[Joshua Lock]

Each failure in the sanity message should be reported on a new line.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 5cf9ea1..39da14b 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -150,7 +150,7 @@ def check_create_long_filename(filepath, pathname):
         if errno == 36: # ENAMETOOLONG
             return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname
         else:
-            return "Failed to create a file in %s: %s" % (pathname, strerror)
+            return "Failed to create a file in %s: %s.\n" % (pathname, strerror)
     return ""
 
 def check_connectivity(d):
-- 
1.7.7.6

[PATCH 5/7] sanity.bbclass: add extra information when SSTATE_CACHE unusable

[Joshua Lock]

If the user does not have write permissions to SSTATE_CACHE, detected by
the check_create_long_filename() test failing with a "Permission denied"
value in strerror, then suggest they might want to use the location as
an entry in SSTATE_MIRRORS.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 39da14b..e86edf7 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -100,6 +100,9 @@ def check_sanity_sstate_dir_change(sstate_dir, data):
     testmsg = ""
     if sstate_dir != "":
         testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR")
+        # If we don't have permissions to SSTATE_DIR, suggest the user set it as an SSTATE_MIRRORS
+        if testmsg.split(': ')[1].strip() == "Permission denied.":
+            testmsg = testmsg + "You could try using %s in SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir)
     return testmsg
 
 def check_sanity_tmpdir_change(tmpdir, data):
-- 
1.7.7.6

[PATCH 6/7] sanity.bbclass: catch an extra exception in check_create_long_filename

[Joshua Lock]

The call to bb.mkdirhier() in check_create_long_filename() can fail with an
OSError, explicitly catch this and report something useful to the user.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index e86edf7..4b52699 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -154,6 +154,8 @@ def check_create_long_filename(filepath, pathname):
             return "Failed to create a file with a long name in %s. Please use a filesystem that does not unreasonably limit filename length.\n" % pathname
         else:
             return "Failed to create a file in %s: %s.\n" % (pathname, strerror)
+    except OSError as (errno, strerror):
+        return "Failed to create %s directory in which to run long name sanity check: %s.\n" % (pathname, strerror)
     return ""
 
 def check_connectivity(d):
-- 
1.7.7.6

[PATCH 7/7] sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed

[Joshua Lock]

This enables a user to use bitbake -e even when the sanity checks are
failing.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 4b52699..689155c 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -539,7 +539,7 @@ def copy_data(e):
 
 addhandler check_sanity_eventhandler
 python check_sanity_eventhandler() {
-    if bb.event.getName(e) == "ConfigParsed" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1":
+    if bb.event.getName(e) == "BuildStarted" and e.data.getVar("BB_WORKERCONTEXT", True) != "1" and e.data.getVar("DISABLE_SANITY_CHECKS", True) != "1":
         sanity_data = copy_data(e)
         check_sanity(sanity_data)
     elif bb.event.getName(e) == "SanityCheck":
-- 
1.7.7.6

Re: [PATCH 0/7] Sanity class clean ups

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2485 bytes --]

This series looks quite sane to me. Nicely done.

-- 
Christopher Larson


On Wednesday, May 23, 2012 at 5:02 PM, Joshua Lock wrote:

> My recent change to the sanity class to warn users when they don't have R/W
> permission to their SSTATE_CACHE directory has caused at least a couple of users
> trouble, including the Yocto project autobuilder.
> 
> I have been unable to reproduce the issue but after discussionwith Elizabeth and
> Chris on #yocto I came up with the following series.
> 
> The series includes a revert my original change as it seems the simple logic has
> severla edge cases. I replace it later in the series with a simple piggy-back
> on the existing check_create_long_filename() call, where I make the suggestion
> of using SSTATE_MIRRORS if that call fails with "Permission denied" when called
> against SSTATE_DIR. This check has been in use for some time and, to the best of
> my knowledge, doesn't trigger invalid failures.
> 
> Thanks to Chris and Elizabeth for pointers as to why this was failing.
> 
> Cheers,
> 
> Joshua
> 
> The following changes since commit e6333825c3482a559a0c0499e17f8f48d3042ddf:
> 
> tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)
> 
> are available in the git repository at:
> git://git.openembedded.org/openembedded-core-contrib (http://git.openembedded.org/openembedded-core-contrib) josh/sanity
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanity
> 
> Joshua Lock (7):
> Revert "sanity.bbclass: check user can read and write to SSTATE_DIR"
> sanity.bbclass: copy the data store and finalise before running
> checks
> sanity.bbclass: data.getVar(VAR, obj, exp) -> obj.getVar(VAR, exp)
> sanity.bbclass: add newline to check_create_long_filename failure
> message
> sanity.bbclass: add extra information when SSTATE_CACHE unusable
> sanity.bbclass: catch an extra exception in
> check_create_long_filename
> sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed
> 
> meta/classes/sanity.bbclass | 153 ++++++++++++++++++++++---------------------
> 1 files changed, 78 insertions(+), 75 deletions(-)
> 
> -- 
> 1.7.7.6
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org (mailto:Openembedded-core@lists.openembedded.org)
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
> 
> 



[-- Attachment #2: Type: text/html, Size: 3732 bytes --]

Re: [PATCH 5/7] sanity.bbclass: add extra information when SSTATE_CACHE unusable

[Joshua Lock]

On 23/05/12 17:03, Joshua Lock wrote:
> If the user does not have write permissions to SSTATE_CACHE, detected by
> the check_create_long_filename() test failing with a "Permission denied"
> value in strerror, then suggest they might want to use the location as
> an entry in SSTATE_MIRRORS.
>
> Signed-off-by: Joshua Lock<josh@linux.intel.com>
> ---
>   meta/classes/sanity.bbclass |    3 +++
>   1 files changed, 3 insertions(+), 0 deletions(-)
>
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index 39da14b..e86edf7 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -100,6 +100,9 @@ def check_sanity_sstate_dir_change(sstate_dir, data):
>       testmsg = ""
>       if sstate_dir != "":
>           testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR")
> +        # If we don't have permissions to SSTATE_DIR, suggest the user set it as an SSTATE_MIRRORS
> +        if testmsg.split(': ')[1].strip() == "Permission denied.":
> +            testmsg = testmsg + "You could try using %s in SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir)
>       return testmsg

Whoops. This one breaks when check_create_long_filename() succeeds. I've 
updated the git repo with the following hunk instead of above:

+        # If we don't have permissions to SSTATE_DIR, suggest the user 
set it as an SSTATE_MIRRORS
+        try:
+            err = testmsg.split(': ')[1].strip()
+            if err == "Permission denied.":
+                testmsg = testmsg + "You could try using %s in 
SSTATE_MIRRORS rather than as an SSTATE_CACHE.\n" % (sstate_dir)
+        except IndexError:
+            pass

Tested with both an SSTATE_DIR I can write to and one I can't.

Thanks,

Joshua
>
>   def check_sanity_tmpdir_change(tmpdir, data):
-- 
Joshua Lock
         Yocto Project
         Intel Open Source Technology Centre

Re: [PATCH 0/7] Sanity class clean ups

[Saul Wold]

On 05/23/2012 05:02 PM, Joshua Lock wrote:
> My recent change to the sanity class to warn users when they don't have R/W
> permission to their SSTATE_CACHE directory has caused at least a couple of users
> trouble, including the Yocto project autobuilder.
>
> I have been unable to reproduce the issue but after discussionwith Elizabeth and
> Chris on #yocto I came up with the following series.
>
> The series includes a revert my original change as it seems the simple logic has
> severla edge cases. I replace it later in the series with a simple piggy-back
> on the existing check_create_long_filename() call, where I make the suggestion
> of using SSTATE_MIRRORS if that call fails with "Permission denied" when called
> against SSTATE_DIR. This check has been in use for some time and, to the best of
> my knowledge, doesn't trigger invalid failures.
>
> Thanks to Chris and Elizabeth for pointers as to why this was failing.
>
> Cheers,
>
> Joshua
>
> The following changes since commit e6333825c3482a559a0c0499e17f8f48d3042ddf:
>
>    tune-mips64.inc: Add new tune file for mips64 big-endian (2012-05-20 20:24:37 -0700)
>
> are available in the git repository at:
>    git://git.openembedded.org/openembedded-core-contrib josh/sanity
>    http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/sanity
>
> Joshua Lock (7):
>    Revert "sanity.bbclass: check user can read and write to SSTATE_DIR"
>    sanity.bbclass: copy the data store and finalise before running
>      checks
>    sanity.bbclass: data.getVar(VAR, obj, exp) ->  obj.getVar(VAR, exp)
>    sanity.bbclass: add newline to check_create_long_filename failure
>      message
>    sanity.bbclass: add extra information when SSTATE_CACHE unusable
>    sanity.bbclass: catch an extra exception in
>      check_create_long_filename
>    sanity.bbclass: check sanity at BuildStarted rather than ConfigParsed
>
>   meta/classes/sanity.bbclass |  153 ++++++++++++++++++++++---------------------
>   1 files changed, 78 insertions(+), 75 deletions(-)
>

Merged the updated version into OE-Core

Thanks
	Sau!