[PATCH 0/1] Initial QA test for reproducible builds

[PATCH 0/1] Initial QA test for reproducible builds

[Joshua Watt]

Implements an initial QA check for reproducible builds. This check is
sufficient for an initial implementation, and will catch a wide variety
of reproducible problems, but it does have the following problems:

 1) It doesn't pass. Currently, about 800 packages fail to build
    in a reproducible manner for core-image-minimal. I've found two
    major sources of non-reproducibility so far:
     a) The perl-module packages don't have a consistent
        SOURCE_DATE_EPOCH which means when they are packaged the
        timestamps on all the files are different. Thankfully, this
        accounts for several hundred of the packages, so fixing this
        should remove a lot of the failures
     b) Debug package strings aren't consistent. It appears that in some
        of the -dbg packages, the linker changes the order of the merged
        .debug_strings section. This trickles down into the packages
        that contain the executables because it changes the hash the
        executable contains to ensure the debug symbols match up.

 2) It's not easy to debug issues when there are reproducibility
    problems. I had initially intended to run diffoscope on the
    resulting files but this takes much longer than I think we are
    willing to run on the autobuilder and also generates far too much
    output to be really useful. I think a better long term route is to
    have the test dump the list of non-reproducible packages and then
    write a helper script that can consumer this list, allow the user to
    select a package, then run diffoscope to examine it.

 3) This test currently is incomplete and won't catch all classes of
    reproducibility problems. At the least, I know that it won't
    consistently catch the use of the __DATE__ macro in source code,
    since that requires the builds to be done on two separate dates (on
    the other hand, use of __TIME__ will be caught pretty reliably since
    the builds are done serially). I suspect the correct solution to
    this is to borrow from Debian and use something like faketime to
    fake out the system time to some suitable future date when doing the
    test build, but this will require some though to how it should be
    implemented.

 4) It currently only tests Debian packages and core-image-minimal. The
    test case has support for building the other package formats and
    other images at the same time, the idea being that the long step in
    this test is building everything from scratch, and building multiple
    package formats and images at the same time will be much faster
    overall than having multiple tests that have to do from-scratch
    builds (although, there might be a way to serialize multiple tests
    and have them share the test build TMPDIR). Until at least 1 package
    format and image are passing, I don't see a huge motivation to
    enable more.

Joshua Watt (1):
  oeqa: Add reproducible build selftest

 meta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++
 1 file changed, 159 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py

-- 
2.21.0

[PATCH 1/1] oeqa: Add reproducible build selftest

[Joshua Watt]

Adds an initial test for reproducible builds to the OE selftest. This
initial test builds core-image-minimal using sstate, then does a clean
build without sstate in another build directory, and finally does a
binary comparison of the resulting package files between the two builds.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++
 1 file changed, 159 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
new file mode 100644
index 00000000000..bf568a4c8ef
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -0,0 +1,159 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright 2019 by Garmin Ltd. or its subsidiaries
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars
+import bisect
+import functools
+import multiprocessing
+import textwrap
+
+MISSING = 'MISSING'
+DIFFERENT = 'DIFFERENT'
+SAME = 'SAME'
+
+@functools.total_ordering
+class CompareResult(object):
+    def __init__(self):
+        self.reference = None
+        self.test = None
+        self.status = 'UNKNOWN'
+
+    def __eq__(self, other):
+        return (self.status, self.test) == (other.status, other.test)
+
+    def __lt__(self, other):
+        return (self.status, self.test) < (other.status, other.test)
+
+class PackageCompareResults(object):
+    def __init__(self):
+        self.total = []
+        self.missing = []
+        self.different = []
+        self.same = []
+
+    def add_result(self, r):
+        self.total.append(r)
+        if r.status == MISSING:
+            self.missing.append(r)
+        elif r.status == DIFFERENT:
+            self.different.append(r)
+        else:
+            self.same.append(r)
+
+    def sort(self):
+        self.total.sort()
+        self.missing.sort()
+        self.different.sort()
+        self.same.sort()
+
+    def __str__(self):
+        return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total))
+
+def compare_file(reference, test, diffutils_sysroot):
+    result = CompareResult()
+    result.reference = reference
+    result.test = test
+
+    if not os.path.exists(reference):
+        result.status = MISSING
+        return result
+
+    r = runCmd(['cmp', '--quiet', reference, test], native_sysroot=diffutils_sysroot, ignore_status=True)
+
+    if r.status:
+        result.status = DIFFERENT
+        return result
+
+    result.status = SAME
+    return result
+
+class ReproducibleTests(OESelftestTestCase):
+    package_classes = ['deb']
+    images = ['core-image-minimal']
+
+    def setUpLocal(self):
+        super().setUpLocal()
+        needed_vars = ['TOPDIR', 'TARGET_PREFIX', 'BB_NUMBER_THREADS']
+        bb_vars = get_bb_vars(needed_vars)
+        for v in needed_vars:
+            setattr(self, v.lower(), bb_vars[v])
+
+        if not hasattr(self.tc, "extraresults"):
+            self.tc.extraresults = {}
+        self.extras = self.tc.extraresults
+
+        self.extras.setdefault('reproducible.rawlogs', {})['log'] = ''
+
+    def append_to_log(self, msg):
+        self.extras['reproducible.rawlogs']['log'] += msg
+
+    def compare_packages(self, reference_dir, test_dir, diffutils_sysroot):
+        result = PackageCompareResults()
+
+        old_cwd = os.getcwd()
+        try:
+            file_result = {}
+            os.chdir(test_dir)
+            with multiprocessing.Pool(processes=int(self.bb_number_threads or 0)) as p:
+                for root, dirs, files in os.walk('.'):
+                    async_result = []
+                    for f in files:
+                        reference_path = os.path.join(reference_dir, root, f)
+                        test_path = os.path.join(test_dir, root, f)
+                        async_result.append(p.apply_async(compare_file, (reference_path, test_path, diffutils_sysroot)))
+
+                    for a in async_result:
+                        result.add_result(a.get())
+
+        finally:
+            os.chdir(old_cwd)
+
+        result.sort()
+        return result
+
+    def test_reproducible_builds(self):
+        capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes]
+
+        common_config = textwrap.dedent('''\
+            INHERIT += "reproducible_build"
+            PACKAGE_CLASSES = "%s"
+            ''') % (' '.join('package_%s' % c for c in self.package_classes))
+
+        # Do an initial build. It's acceptable for this build to use sstate
+        self.write_config(common_config)
+        vars_reference = get_bb_vars(capture_vars)
+        bitbake(' '.join(self.images))
+
+        # Build native utilities
+        bitbake("diffutils-native -c addto_recipe_sysroot")
+        diffutils_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffutils-native")
+
+        # Perform another build. This build should *not* share sstate or pull
+        # from any mirrors, but sharing a DL_DIR is fine
+        self.write_config(textwrap.dedent('''\
+            TMPDIR = "${TOPDIR}/reproducible/tmp"
+            SSTATE_DIR = "${TMPDIR}/sstate"
+            SSTATE_MIRROR = ""
+            ''') + common_config)
+        vars_test = get_bb_vars(capture_vars)
+        bitbake(' '.join(self.images))
+
+        for c in self.package_classes:
+            package_class = 'package_' + c
+
+            deploy_reference = vars_reference['DEPLOY_DIR_' + c.upper()]
+            deploy_test = vars_test['DEPLOY_DIR_' + c.upper()]
+
+            result = self.compare_packages(deploy_reference, deploy_test, diffutils_sysroot)
+
+            self.logger.info('Reproducibility summary for %s: %s' % (c, result))
+
+            self.append_to_log('\n'.join("%s: %s" % (r.status, r.test) for r in result.total))
+
+            if result.missing or result.different:
+                self.fail("The following %s packages are missing or different: %s" %
+                        (c, ' '.join(r.test for r in (result.missing + result.different))))
+
-- 
2.21.0

Re: [PATCH 0/1] Initial QA test for reproducible builds

[Khem Raj]

Hi Joshua

Thanks for contributing this will provide some teeth to reproducible 
builds QA

On 5/20/19 9:57 AM, Joshua Watt wrote:
> Implements an initial QA check for reproducible builds. This check is
> sufficient for an initial implementation, and will catch a wide variety
> of reproducible problems, but it does have the following problems:
> 
>   1) It doesn't pass. Currently, about 800 packages fail to build
>      in a reproducible manner for core-image-minimal. I've found two
>      major sources of non-reproducibility so far:
>       a) The perl-module packages don't have a consistent
>          SOURCE_DATE_EPOCH which means when they are packaged the
>          timestamps on all the files are different. Thankfully, this
>          accounts for several hundred of the packages, so fixing this
>          should remove a lot of the failures

maybe we can start with inhriting reproducible_build_simple which has 
hardcoded values for SOURCE_DATE_EPOCH

>       b) Debug package strings aren't consistent. It appears that in some
>          of the -dbg packages, the linker changes the order of the merged
>          .debug_strings section. This trickles down into the packages
>          that contain the executables because it changes the hash the
>          executable contains to ensure the debug symbols match up.
> 

try adding -fno-merge-debug-strings to linker and see if that fixes this 
problem. If that happens then we know its an option to add when doing 
reproducible builds.

>   2) It's not easy to debug issues when there are reproducibility
>      problems. I had initially intended to run diffoscope on the
>      resulting files but this takes much longer than I think we are
>      willing to run on the autobuilder and also generates far too much
>      output to be really useful. I think a better long term route is to
>      have the test dump the list of non-reproducible packages and then
>      write a helper script that can consumer this list, allow the user to
>      select a package, then run diffoscope to examine it.

I think that might be needed to wrap diffoscope.

> 
>   3) This test currently is incomplete and won't catch all classes of
>      reproducibility problems. At the least, I know that it won't
>      consistently catch the use of the __DATE__ macro in source code,
>      since that requires the builds to be done on two separate dates (on
>      the other hand, use of __TIME__ will be caught pretty reliably since
>      the builds are done serially). I suspect the correct solution to
>      this is to borrow from Debian and use something like faketime to
>      fake out the system time to some suitable future date when doing the
>      test build, but this will require some though to how it should be
>      implemented.
> 
>   4) It currently only tests Debian packages and core-image-minimal. The
>      test case has support for building the other package formats and
>      other images at the same time, the idea being that the long step in
>      this test is building everything from scratch, and building multiple
>      package formats and images at the same time will be much faster
>      overall than having multiple tests that have to do from-scratch
>      builds (although, there might be a way to serialize multiple tests
>      and have them share the test build TMPDIR). Until at least 1 package
>      format and image are passing, I don't see a huge motivation to
>      enable more.

why does it have to depend on packaging backend ?

> 
> Joshua Watt (1):
>    oeqa: Add reproducible build selftest
> 
>   meta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++
>   1 file changed, 159 insertions(+)
>   create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py
> 

Re: [PATCH 0/1] Initial QA test for reproducible builds

[Joshua Watt]

On Mon, 2019-05-20 at 10:47 -0700, Khem Raj wrote:
> Hi Joshua
> 
> Thanks for contributing this will provide some teeth to reproducible 
> builds QA
> 
> On 5/20/19 9:57 AM, Joshua Watt wrote:
> > Implements an initial QA check for reproducible builds. This check
> > is
> > sufficient for an initial implementation, and will catch a wide
> > variety
> > of reproducible problems, but it does have the following problems:
> > 
> >   1) It doesn't pass. Currently, about 800 packages fail to build
> >      in a reproducible manner for core-image-minimal. I've found
> > two
> >      major sources of non-reproducibility so far:
> >       a) The perl-module packages don't have a consistent
> >          SOURCE_DATE_EPOCH which means when they are packaged the
> >          timestamps on all the files are different. Thankfully,
> > this
> >          accounts for several hundred of the packages, so fixing
> > this
> >          should remove a lot of the failures
> 
> maybe we can start with inhriting reproducible_build_simple which
> has 
> hardcoded values for SOURCE_DATE_EPOCH

reproducible_build.bbclass automatically inherits
reproducible_build_simple if BUILD_REPRODUCIBLE_BINARIES is set, and it
sets that by default.

I'm not actually sure how the value of SOURCE_DATE_EPOCH set in
reproducible_build_simple.bbclass matters at all. AFAIK,
reproducible_build.bbclass tries *really* hard to get a discernible
date from the source code repository itself after do_unpack, and
failing that uses 0. Near as I can tell, the value set in the class or
by the user only matters until the end of do_unpack.

I think something happens to be broken with this heuristic in the perl
recipe. I haven't tracked it down just yet.

> 
> >       b) Debug package strings aren't consistent. It appears that
> > in some
> >          of the -dbg packages, the linker changes the order of the
> > merged
> >          .debug_strings section. This trickles down into the
> > packages
> >          that contain the executables because it changes the hash
> > the
> >          executable contains to ensure the debug symbols match up.
> > 
> 
> try adding -fno-merge-debug-strings to linker and see if that fixes
> this 
> problem. If that happens then we know its an option to add when
> doing 
> reproducible builds.

Excellent. I will try that.

> 
> >   2) It's not easy to debug issues when there are reproducibility
> >      problems. I had initially intended to run diffoscope on the
> >      resulting files but this takes much longer than I think we are
> >      willing to run on the autobuilder and also generates far too
> > much
> >      output to be really useful. I think a better long term route
> > is to
> >      have the test dump the list of non-reproducible packages and
> > then
> >      write a helper script that can consumer this list, allow the
> > user to
> >      select a package, then run diffoscope to examine it.
> 
> I think that might be needed to wrap diffoscope.

I'm not sure I quite follow what you are saying here?

> 
> >   3) This test currently is incomplete and won't catch all classes
> > of
> >      reproducibility problems. At the least, I know that it won't
> >      consistently catch the use of the __DATE__ macro in source
> > code,
> >      since that requires the builds to be done on two separate
> > dates (on
> >      the other hand, use of __TIME__ will be caught pretty reliably
> > since
> >      the builds are done serially). I suspect the correct solution
> > to
> >      this is to borrow from Debian and use something like faketime
> > to
> >      fake out the system time to some suitable future date when
> > doing the
> >      test build, but this will require some though to how it should
> > be
> >      implemented.
> > 
> >   4) It currently only tests Debian packages and core-image-
> > minimal. The
> >      test case has support for building the other package formats
> > and
> >      other images at the same time, the idea being that the long
> > step in
> >      this test is building everything from scratch, and building
> > multiple
> >      package formats and images at the same time will be much
> > faster
> >      overall than having multiple tests that have to do from-
> > scratch
> >      builds (although, there might be a way to serialize multiple
> > tests
> >      and have them share the test build TMPDIR). Until at least 1
> > package
> >      format and image are passing, I don't see a huge motivation to
> >      enable more.
> 
> why does it have to depend on packaging backend ?

It doesn't particularly. Comparing the end packages is just the easiest
way to check for reproducibility (I think?). I originally tried
comparing sstate tarballs, but I quickly realized that was going to be
very difficult without making a lot of changes to the way sstate
tarballs are generated (e.g. timestamps and such).

On the other end of the spectrum would be to compare the final image
files themselves; I like that idea and I think we *should* do that
after we get the packages to be reproducible, but it's not very useful
right now because it's hard(er) to track a difference in the image back
to a recipe than it is to track a difference in a package back.

You could leave the test up to whatever package classes are enabled on
the autobuilder, but I think this is less than ideal right now because:

 1) IPKs aren't reproducible *at all*... they are basically all
different. Perhaps because of timestamps? Maybe that's easy fix?

 2) The long pull on this QA is the test build from scratch... it takes
quite a while and will take even longer with a bigger image.
Comparatively, generating multiple packages classes doesn't add that
much more time to the build, so I *think* it might make sense in the
long run to have the test build all the package formats in one go, or
at least being able to share the test build TMPDIR between multiple
serial tests.

> 
> > Joshua Watt (1):
> >    oeqa: Add reproducible build selftest
> > 
> >   meta/lib/oeqa/selftest/cases/reproducible.py | 159
> > +++++++++++++++++++
> >   1 file changed, 159 insertions(+)
> >   create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py
> > 
-- 
Joshua Watt <JPEWhacker@gmail.com>

[PATCH v2] oeqa: Add reproducible build selftest

[Joshua Watt]

Adds an initial test for reproducible builds to the OE selftest. This
initial test builds core-image-minimal using sstate, then does a clean
build without sstate in another build directory, and finally does a
binary comparison of the resulting package files between the two builds.

The test is currently always skipped since it doesn't pass yet, but it
can easily be enabled locally

Signed-off-by: Joshua Watt <JPEWHacker@gmail.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 160 +++++++++++++++++++
 1 file changed, 160 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
new file mode 100644
index 00000000000..6dc83d28474
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -0,0 +1,160 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright 2019 by Garmin Ltd. or its subsidiaries
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars
+import functools
+import multiprocessing
+import textwrap
+import unittest
+
+MISSING = 'MISSING'
+DIFFERENT = 'DIFFERENT'
+SAME = 'SAME'
+
+@functools.total_ordering
+class CompareResult(object):
+    def __init__(self):
+        self.reference = None
+        self.test = None
+        self.status = 'UNKNOWN'
+
+    def __eq__(self, other):
+        return (self.status, self.test) == (other.status, other.test)
+
+    def __lt__(self, other):
+        return (self.status, self.test) < (other.status, other.test)
+
+class PackageCompareResults(object):
+    def __init__(self):
+        self.total = []
+        self.missing = []
+        self.different = []
+        self.same = []
+
+    def add_result(self, r):
+        self.total.append(r)
+        if r.status == MISSING:
+            self.missing.append(r)
+        elif r.status == DIFFERENT:
+            self.different.append(r)
+        else:
+            self.same.append(r)
+
+    def sort(self):
+        self.total.sort()
+        self.missing.sort()
+        self.different.sort()
+        self.same.sort()
+
+    def __str__(self):
+        return 'same=%i different=%i missing=%i total=%i' % (len(self.same), len(self.different), len(self.missing), len(self.total))
+
+def compare_file(reference, test, diffutils_sysroot):
+    result = CompareResult()
+    result.reference = reference
+    result.test = test
+
+    if not os.path.exists(reference):
+        result.status = MISSING
+        return result
+
+    r = runCmd(['cmp', '--quiet', reference, test], native_sysroot=diffutils_sysroot, ignore_status=True)
+
+    if r.status:
+        result.status = DIFFERENT
+        return result
+
+    result.status = SAME
+    return result
+
+class ReproducibleTests(OESelftestTestCase):
+    package_classes = ['deb']
+    images = ['core-image-minimal']
+
+    def setUpLocal(self):
+        super().setUpLocal()
+        needed_vars = ['TOPDIR', 'TARGET_PREFIX', 'BB_NUMBER_THREADS']
+        bb_vars = get_bb_vars(needed_vars)
+        for v in needed_vars:
+            setattr(self, v.lower(), bb_vars[v])
+
+        if not hasattr(self.tc, "extraresults"):
+            self.tc.extraresults = {}
+        self.extras = self.tc.extraresults
+
+        self.extras.setdefault('reproducible.rawlogs', {})['log'] = ''
+
+    def append_to_log(self, msg):
+        self.extras['reproducible.rawlogs']['log'] += msg
+
+    def compare_packages(self, reference_dir, test_dir, diffutils_sysroot):
+        result = PackageCompareResults()
+
+        old_cwd = os.getcwd()
+        try:
+            file_result = {}
+            os.chdir(test_dir)
+            with multiprocessing.Pool(processes=int(self.bb_number_threads or 0)) as p:
+                for root, dirs, files in os.walk('.'):
+                    async_result = []
+                    for f in files:
+                        reference_path = os.path.join(reference_dir, root, f)
+                        test_path = os.path.join(test_dir, root, f)
+                        async_result.append(p.apply_async(compare_file, (reference_path, test_path, diffutils_sysroot)))
+
+                    for a in async_result:
+                        result.add_result(a.get())
+
+        finally:
+            os.chdir(old_cwd)
+
+        result.sort()
+        return result
+
+    @unittest.skip("Reproducible builds do not yet pass")
+    def test_reproducible_builds(self):
+        capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes]
+
+        common_config = textwrap.dedent('''\
+            INHERIT += "reproducible_build"
+            PACKAGE_CLASSES = "%s"
+            ''') % (' '.join('package_%s' % c for c in self.package_classes))
+
+        # Do an initial build. It's acceptable for this build to use sstate
+        self.write_config(common_config)
+        vars_reference = get_bb_vars(capture_vars)
+        bitbake(' '.join(self.images))
+
+        # Build native utilities
+        bitbake("diffutils-native -c addto_recipe_sysroot")
+        diffutils_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "diffutils-native")
+
+        # Perform another build. This build should *not* share sstate or pull
+        # from any mirrors, but sharing a DL_DIR is fine
+        self.write_config(textwrap.dedent('''\
+            TMPDIR = "${TOPDIR}/reproducible/tmp"
+            SSTATE_DIR = "${TMPDIR}/sstate"
+            SSTATE_MIRROR = ""
+            ''') + common_config)
+        vars_test = get_bb_vars(capture_vars)
+        bitbake(' '.join(self.images))
+
+        for c in self.package_classes:
+            package_class = 'package_' + c
+
+            deploy_reference = vars_reference['DEPLOY_DIR_' + c.upper()]
+            deploy_test = vars_test['DEPLOY_DIR_' + c.upper()]
+
+            result = self.compare_packages(deploy_reference, deploy_test, diffutils_sysroot)
+
+            self.logger.info('Reproducibility summary for %s: %s' % (c, result))
+
+            self.append_to_log('\n'.join("%s: %s" % (r.status, r.test) for r in result.total))
+
+            if result.missing or result.different:
+                self.fail("The following %s packages are missing or different: %s" %
+                        (c, ' '.join(r.test for r in (result.missing + result.different))))
+
-- 
2.21.0

✗ patchtest: failure for Initial QA test for reproducible builds (rev2)

[Patchwork]

== Series Details ==

Series: Initial QA test for reproducible builds (rev2)
Revision: 2
URL   : https://patchwork.openembedded.org/series/17720/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Errors in your Python code were encountered [test_pylint] 
  Suggested fix    Correct the lines introduced by your patch
  Output           Please, fix the listed issues:
                   meta/lib/oeqa/selftest/cases/reproducible.py does not exist



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe