[PATCH 1/4] recipetool: Set master branch only as fallback

[PATCH 1/4] recipetool: Set master branch only as fallback

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)'
sets the branch= parameter too early to master and thereby breaks the
-B/--srcbranch option.

ERROR: branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

 scripts/lib/recipetool/create.py | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 4f6e01c639..406c97f1c5 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -389,9 +389,6 @@ def reformat_git_uri(uri):
                 parms.update({('protocol', 'ssh')})
         elif (scheme == "http" or scheme == 'https' or scheme == 'ssh') and not ('protocol' in parms):
             parms.update({('protocol', scheme)})
-        # We assume 'master' branch if not set
-        if not 'branch' in parms:
-            parms.update({('branch', 'master')})
         # Always append 'git://'
         fUrl = bb.fetch2.encodeurl(('git', host, path, user, pswd, parms))
         return fUrl
@@ -481,6 +478,9 @@ def create_recipe(args):
             storeTagName = params['tag']
             params['nobranch'] = '1'
             del params['tag']
+        # Assume 'master' branch if not set
+        if scheme in ['git', 'gitsm'] and 'branch' not in params and 'nobranch' not in params:
+            params['branch'] = 'master'
         fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
@@ -530,10 +530,9 @@ def create_recipe(args):
             # Remove HEAD reference point and drop remote prefix
             get_branch = [x.split('/', 1)[1] for x in get_branch if not x.startswith('origin/HEAD')]
             if 'master' in get_branch:
-                # If it is master, we do not need to append 'branch=master' as this is default.
                 # Even with the case where get_branch has multiple objects, if 'master' is one
                 # of them, we should default take from 'master'
-                srcbranch = ''
+                srcbranch = 'master'
             elif len(get_branch) == 1:
                 # If 'master' isn't in get_branch and get_branch contains only ONE object, then store result into 'srcbranch'
                 srcbranch = get_branch[0]
@@ -546,8 +545,8 @@ def create_recipe(args):
         # Since we might have a value in srcbranch, we need to
         # recontruct the srcuri to include 'branch' in params.
         scheme, network, path, user, passwd, params = bb.fetch2.decodeurl(srcuri)
-        if srcbranch:
-            params['branch'] = srcbranch
+        if scheme in ['git', 'gitsm']:
+            params['branch'] = srcbranch or 'master'
 
         if storeTagName and scheme in ['git', 'gitsm']:
             # Check srcrev using tag and check validity of the tag
@@ -606,7 +605,7 @@ def create_recipe(args):
                     splitline = line.split()
                     if len(splitline) > 1:
                         if splitline[0] == 'origin' and scriptutils.is_src_url(splitline[1]):
-                            srcuri = reformat_git_uri(splitline[1])
+                            srcuri = reformat_git_uri(splitline[1]) + ';branch=master'
                             srcsubdir = 'git'
                             break
 
-- 
2.20.1

[PATCH 2/4] selftest/recipetool: Split tests into separate test classes

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Split tests into separate test classes to speed up individual test runs
by reducing the test setup to a minimum.

The pkgdata generation is only needed for the append tests and slow down
the other tests.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

 meta/lib/oeqa/selftest/cases/recipetool.py | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index 8dc00ac10e..1aedc02b99 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -68,7 +68,7 @@ class RecipetoolBase(devtool.DevtoolBase):
         return bbappendfile, result.output
 
 
-class RecipetoolTests(RecipetoolBase):
+class RecipetoolAppendTests(RecipetoolBase):
 
     @classmethod
     def setUpClass(cls):
@@ -76,9 +76,8 @@ class RecipetoolTests(RecipetoolBase):
         # Ensure we have the right data in shlibs/pkgdata
         cls.logger.info('Running bitbake to generate pkgdata')
         bitbake('-c packagedata base-files coreutils busybox selftest-recipetool-appendfile')
-        bb_vars = get_bb_vars(['COREBASE', 'BBPATH'])
+        bb_vars = get_bb_vars(['COREBASE'])
         cls.corebase = bb_vars['COREBASE']
-        cls.bbpath = bb_vars['BBPATH']
 
     def _try_recipetool_appendfile(self, testrecipe, destfile, newfile, options, expectedlines, expectedfiles):
         cmd = 'recipetool appendfile %s %s %s %s' % (self.templayerdir, destfile, newfile, options)
@@ -332,6 +331,9 @@ class RecipetoolTests(RecipetoolBase):
         filename = try_appendfile_wc('-w')
         self.assertEqual(filename, recipefn.split('_')[0] + '_%.bbappend')
 
+
+class RecipetoolCreateTests(RecipetoolBase):
+
     def test_recipetool_create(self):
         # Try adding a recipe
         tempsrc = os.path.join(self.tempdir, 'srctree')
@@ -518,6 +520,15 @@ class RecipetoolTests(RecipetoolBase):
         inherits = ['pkgconfig', 'autotools']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+
+class RecipetoolTests(RecipetoolBase):
+
+    @classmethod
+    def setUpClass(cls):
+        super(RecipetoolTests, cls).setUpClass()
+        bb_vars = get_bb_vars(['BBPATH'])
+        cls.bbpath = bb_vars['BBPATH']
+
     def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
         dstdir = basedstdir
         self.assertTrue(os.path.exists(dstdir))
-- 
2.20.1

[PATCH 3/4] selftest/recipetool: Add tests for branch parameter and srcbranch option

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

The recipetool support two ways to pass a branch and fallback to master
if no branch is defined. Add tests for default branch, branch parameter
and srcbranch option.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---
This commit changes the test repository from matchbox-terminal to
matchbox-keyboard because matchbox-terminal has only one master branch.
Maybe the test should use a special test repository.

 meta/lib/oeqa/selftest/cases/recipetool.py | 38 +++++++++++++++-------
 1 file changed, 27 insertions(+), 11 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index 1aedc02b99..b77a2712f5 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -350,7 +350,7 @@ class RecipetoolCreateTests(RecipetoolBase):
         checkvars['SRC_URI[sha256sum]'] = '2e6a401cac9024db2288297e3be1a8ab60e7401ba8e91225218aaf4a27e82a07'
         self._test_recipe_contents(recipefile, checkvars, [])
 
-    def test_recipetool_create_git(self):
+    def test_recipetool_create_autotools(self):
         if 'x11' not in get_bb_var('DISTRO_FEATURES'):
             self.skipTest('Test requires x11 as distro feature')
         # Ensure we have the right data in shlibs/pkgdata
@@ -359,7 +359,7 @@ class RecipetoolCreateTests(RecipetoolBase):
         tempsrc = os.path.join(self.tempdir, 'srctree')
         os.makedirs(tempsrc)
         recipefile = os.path.join(self.tempdir, 'libmatchbox.bb')
-        srcuri = 'git://git.yoctoproject.org/libmatchbox;branch=master'
+        srcuri = 'git://git.yoctoproject.org/libmatchbox'
         result = runCmd(['recipetool', 'create', '-o', recipefile, srcuri + ";rev=9f7cf8895ae2d39c465c04cc78e918c157420269", '-x', tempsrc])
         self.assertTrue(os.path.isfile(recipefile), 'recipetool did not create recipe file; output:\n%s' % result.output)
         checkvars = {}
@@ -367,7 +367,7 @@ class RecipetoolCreateTests(RecipetoolBase):
         checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING;md5=7fbc338309ac38fefcd64b04bb903e34'
         checkvars['S'] = '${WORKDIR}/git'
         checkvars['PV'] = '1.11+git${SRCPV}'
-        checkvars['SRC_URI'] = srcuri
+        checkvars['SRC_URI'] = srcuri + ';branch=master'
         checkvars['DEPENDS'] = set(['libcheck', 'libjpeg-turbo', 'libpng', 'libx11', 'libxext', 'pango'])
         inherits = ['autotools', 'pkgconfig']
         self._test_recipe_contents(recipefile, checkvars, inherits)
@@ -506,19 +506,35 @@ class RecipetoolCreateTests(RecipetoolBase):
         inherits = ['setuptools3']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
-    def test_recipetool_create_git_http(self):
+    def _test_recipetool_create_git(self, srcuri, branch=None):
         # Basic test to check http git URL mangling works
         temprecipe = os.path.join(self.tempdir, 'recipe')
         os.makedirs(temprecipe)
-        recipefile = os.path.join(temprecipe, 'matchbox-terminal_git.bb')
-        srcuri = 'http://git.yoctoproject.org/git/matchbox-terminal'
-        result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
+        name = srcuri.split(';')[0].split('/')[-1]
+        recipefile = os.path.join(temprecipe, name + '_git.bb')
+        options = ' -B %s' % branch if branch else ''
+        result = runCmd('recipetool create -o %s%s "%s"' % (temprecipe, options, srcuri))
         self.assertTrue(os.path.isfile(recipefile))
         checkvars = {}
-        checkvars['LICENSE'] = set(['GPLv2'])
-        checkvars['SRC_URI'] = 'git://git.yoctoproject.org/git/matchbox-terminal;protocol=http;branch=master'
-        inherits = ['pkgconfig', 'autotools']
-        self._test_recipe_contents(recipefile, checkvars, inherits)
+        checkvars['SRC_URI'] = srcuri
+        for scheme in ['http', 'https']:
+            if srcuri.startswith(scheme + ":"):
+                checkvars['SRC_URI'] = 'git%s;protocol=%s' % (srcuri[len(scheme):], scheme)
+        if ';branch=' not in srcuri:
+            checkvars['SRC_URI'] += ';branch=' + (branch or 'master')
+        self._test_recipe_contents(recipefile, checkvars, [])
+
+    def test_recipetool_create_git_http(self):
+        self._test_recipetool_create_git('http://git.yoctoproject.org/git/matchbox-keyboard')
+
+    def test_recipetool_create_git_srcuri_master(self):
+        self._test_recipetool_create_git('git://git.yoctoproject.org/matchbox-keyboard;branch=master')
+
+    def test_recipetool_create_git_srcuri_branch(self):
+        self._test_recipetool_create_git('git://git.yoctoproject.org/matchbox-keyboard;branch=matchbox-keyboard-0-1')
+
+    def test_recipetool_create_git_srcbranch(self):
+        self._test_recipetool_create_git('git://git.yoctoproject.org/matchbox-keyboard', 'matchbox-keyboard-0-1')
 
 
 class RecipetoolTests(RecipetoolBase):
-- 
2.20.1

[PATCH 4/4] selftest: devtool: Separate common functions and devtool sstate setup into two classes

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

The selftest recipetool base class reuse the selftest devtool base
class. Thereby the selftest devtool base class setup its own devtool
sstate and the selftest recipetool classes trigger the build of recipes.
This leads to the problem that the build artifacts doesn't reach the
persistent sstate cache and rebuild on every selftest run.

Move the common selftest devtool functions into its own class and use
the sstate cache in the recipetool tests.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

 meta/lib/oeqa/selftest/cases/devtool.py    | 45 +++++++++++++---------
 meta/lib/oeqa/selftest/cases/recipetool.py |  4 +-
 2 files changed, 28 insertions(+), 21 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index d2b31af80e..96f40ac9a0 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -80,32 +80,15 @@ def tearDownModule():
         bb.utils.edit_bblayers_conf(bblayers_conf, None, None, bblayers_edit_cb)
     shutil.rmtree(templayerdir)
 
-class DevtoolBase(OESelftestTestCase):
-
-    @classmethod
-    def setUpClass(cls):
-        super(DevtoolBase, cls).setUpClass()
-        bb_vars = get_bb_vars(['TOPDIR', 'SSTATE_DIR'])
-        cls.original_sstate = bb_vars['SSTATE_DIR']
-        cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
-        cls.sstate_conf  = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
-        cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
-                            % cls.original_sstate)
-
-    @classmethod
-    def tearDownClass(cls):
-        cls.logger.debug('Deleting devtool sstate cache on %s' % cls.devtool_sstate)
-        runCmd('rm -rf %s' % cls.devtool_sstate)
-        super(DevtoolBase, cls).tearDownClass()
+class DevtoolTestCase(OESelftestTestCase):
 
     def setUp(self):
         """Test case setup function"""
-        super(DevtoolBase, self).setUp()
+        super(DevtoolTestCase, self).setUp()
         self.workspacedir = os.path.join(self.builddir, 'workspace')
         self.assertTrue(not os.path.exists(self.workspacedir),
                         'This test cannot be run with a workspace directory '
                         'under the build directory')
-        self.append_config(self.sstate_conf)
 
     def _check_src_repo(self, repo_dir):
         """Check srctree git repository"""
@@ -236,6 +219,30 @@ class DevtoolBase(OESelftestTestCase):
         return filelist
 
 
+class DevtoolBase(DevtoolTestCase):
+
+    @classmethod
+    def setUpClass(cls):
+        super(DevtoolBase, cls).setUpClass()
+        bb_vars = get_bb_vars(['TOPDIR', 'SSTATE_DIR'])
+        cls.original_sstate = bb_vars['SSTATE_DIR']
+        cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
+        cls.sstate_conf  = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
+        cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
+                            % cls.original_sstate)
+
+    @classmethod
+    def tearDownClass(cls):
+        cls.logger.debug('Deleting devtool sstate cache on %s' % cls.devtool_sstate)
+        runCmd('rm -rf %s' % cls.devtool_sstate)
+        super(DevtoolBase, cls).tearDownClass()
+
+    def setUp(self):
+        """Test case setup function"""
+        super(DevtoolBase, self).setUp()
+        self.append_config(self.sstate_conf)
+
+
 class DevtoolTests(DevtoolBase):
 
     def test_create_workspace(self):
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index b77a2712f5..439e41597c 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -25,7 +25,7 @@ def tearDownModule():
     runCmd('rm -rf %s' % templayerdir)
 
 
-class RecipetoolBase(devtool.DevtoolBase):
+class RecipetoolBase(devtool.DevtoolTestCase):
 
     def setUpLocal(self):
         super(RecipetoolBase, self).setUpLocal()
@@ -72,7 +72,7 @@ class RecipetoolAppendTests(RecipetoolBase):
 
     @classmethod
     def setUpClass(cls):
-        super(RecipetoolTests, cls).setUpClass()
+        super(RecipetoolAppendTests, cls).setUpClass()
         # Ensure we have the right data in shlibs/pkgdata
         cls.logger.info('Running bitbake to generate pkgdata')
         bitbake('-c packagedata base-files coreutils busybox selftest-recipetool-appendfile')
-- 
2.20.1

Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback

[Richard Purdie]

On Mon, 2021-12-06 at 09:04 +0100, Stefan Herbrechtsmeier wrote:
> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
> 
> The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)'
> sets the branch= parameter too early to master and thereby breaks the
> -B/--srcbranch option.
> 
> ERROR: branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other
> 
> Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
> ---
> 
>  scripts/lib/recipetool/create.py | 15 +++++++--------
>  1 file changed, 7 insertions(+), 8 deletions(-)

I think something in this series is causing:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2926/steps/14/logs/stdio

(the other selftest builds also failed)

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback

[Stefan Herbrechtsmeier]

Hi Richard,

Am 07.12.2021 um 16:45 schrieb Richard Purdie:
> On Mon, 2021-12-06 at 09:04 +0100, Stefan Herbrechtsmeier wrote:
>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>>
>> The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)'
>> sets the branch= parameter too early to master and thereby breaks the
>> -B/--srcbranch option.
>>
>> ERROR: branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other
>>
>> Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>> ---
>>
>>   scripts/lib/recipetool/create.py | 15 +++++++--------
>>   1 file changed, 7 insertions(+), 8 deletions(-)
> 
> I think something in this series is causing:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2926/steps/14/logs/stdio

Sorry I missed the adaption of the test test_devtool_add_fetch_git. It 
doesn't expect a "branch=master" which is wrong.

- gitsm://git.yoctoproject.org/mraa;branch=master
?                                  --------------
+ gitsm://git.yoctoproject.org/mraa

I will update my patch series but need some time to test it locally. The 
devtool test build a lot of recipe on every run because it use a private 
sstate cache. How does the autobuilder overcome this problem?

> 
> (the other selftest builds also failed)
What do you mean by this?

Regards
   Stefan

Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

On Wed, 8 Dec 2021 at 08:45, Stefan Herbrechtsmeier <
stefan.herbrechtsmeier-oss@weidmueller.com> wrote:

> I will update my patch series but need some time to test it locally. The
> devtool test build a lot of recipe on every run because it use a private
> sstate cache. How does the autobuilder overcome this problem?
>

If you don't rebase the branch used for testing, and set the cache to a
directory outside of build directory (e.g. /home/stefan/sstate ) it will
reuse that cache between selftest invocations.

The autobuilder uses a single writeable cache for all builds and builders,
mounted as read-write NFS partition.

Alex

[-- Attachment #2: Type: text/html, Size: 1066 bytes --]

Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback

[Stefan Herbrechtsmeier]

Am 08.12.2021 um 09:10 schrieb Alexander Kanavin:
> On Wed, 8 Dec 2021 at 08:45, Stefan Herbrechtsmeier 
> <stefan.herbrechtsmeier-oss@weidmueller.com 
> <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
> 
>     I will update my patch series but need some time to test it locally.
>     The
>     devtool test build a lot of recipe on every run because it use a
>     private
>     sstate cache. How does the autobuilder overcome this problem?
> 
> 
> If you don't rebase the branch used for testing, and set the cache to a 
> directory outside of build directory (e.g. /home/stefan/sstate ) it will 
> reuse that cache between selftest invocations.

The devtool test use its own PRIVATE sstate cache:

def setUpClass(cls):
...
cls.original_sstate = bb_vars['SSTATE_DIR']
cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
cls.sstate_conf  = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
                     % cls.original_sstate)

And remove it after the test run:

def tearDownClass(cls):
...
runCmd('rm -rf %s' % cls.devtool_sstate)

Even a keep builddir doesn't help to keep the private sstate cache.

> The autobuilder uses a single writeable cache for all builds and 
> builders, mounted as read-write NFS partition.

This doesn't help in this case. The devtool test clean parts of the 
sstate cache and therefore use its private sstate cache and the original 
sstate cache as mirror. This is a problem if a test triggers a build of 
dependencies.

Regards
   Stefan

Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1884 bytes --]

I haven't looked at this, but probably the test could be adjusted to first
build the needed parts with only the original cache?

Alex

On Wed, 8 Dec 2021 at 09:27, Stefan Herbrechtsmeier <
stefan.herbrechtsmeier-oss@weidmueller.com> wrote:

> Am 08.12.2021 um 09:10 schrieb Alexander Kanavin:
> > On Wed, 8 Dec 2021 at 08:45, Stefan Herbrechtsmeier
> > <stefan.herbrechtsmeier-oss@weidmueller.com
> > <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
> >
> >     I will update my patch series but need some time to test it locally.
> >     The
> >     devtool test build a lot of recipe on every run because it use a
> >     private
> >     sstate cache. How does the autobuilder overcome this problem?
> >
> >
> > If you don't rebase the branch used for testing, and set the cache to a
> > directory outside of build directory (e.g. /home/stefan/sstate ) it will
> > reuse that cache between selftest invocations.
>
> The devtool test use its own PRIVATE sstate cache:
>
> def setUpClass(cls):
> ...
> cls.original_sstate = bb_vars['SSTATE_DIR']
> cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
> cls.sstate_conf  = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
> cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
>                      % cls.original_sstate)
>
> And remove it after the test run:
>
> def tearDownClass(cls):
> ...
> runCmd('rm -rf %s' % cls.devtool_sstate)
>
> Even a keep builddir doesn't help to keep the private sstate cache.
>
> > The autobuilder uses a single writeable cache for all builds and
> > builders, mounted as read-write NFS partition.
>
> This doesn't help in this case. The devtool test clean parts of the
> sstate cache and therefore use its private sstate cache and the original
> sstate cache as mirror. This is a problem if a test triggers a build of
> dependencies.
>
> Regards
>    Stefan
>

[-- Attachment #2: Type: text/html, Size: 2706 bytes --]

Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback

[Stefan Herbrechtsmeier]

Am 08.12.2021 um 09:32 schrieb Alexander Kanavin via lists.openembedded.org:
> I haven't looked at this, but probably the test could be adjusted to 
> first build the needed parts with only the original cache?

This needs a rework of the tests because the private sstate cache is set 
in the test setup and we need to know the dependencies of the test to 
build those manual.

> On Wed, 8 Dec 2021 at 09:27, Stefan Herbrechtsmeier 
> <stefan.herbrechtsmeier-oss@weidmueller.com 
> <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>> wrote:
> 
>     Am 08.12.2021 um 09:10 schrieb Alexander Kanavin:
>      > On Wed, 8 Dec 2021 at 08:45, Stefan Herbrechtsmeier
>      > <stefan.herbrechtsmeier-oss@weidmueller.com
>     <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>
>      > <mailto:stefan.herbrechtsmeier-oss@weidmueller.com
>     <mailto:stefan.herbrechtsmeier-oss@weidmueller.com>>> wrote:
>      >
>      >     I will update my patch series but need some time to test it
>     locally.
>      >     The
>      >     devtool test build a lot of recipe on every run because it use a
>      >     private
>      >     sstate cache. How does the autobuilder overcome this problem?
>      >
>      >
>      > If you don't rebase the branch used for testing, and set the
>     cache to a
>      > directory outside of build directory (e.g. /home/stefan/sstate )
>     it will
>      > reuse that cache between selftest invocations.
> 
>     The devtool test use its own PRIVATE sstate cache:
> 
>     def setUpClass(cls):
>     ...
>     cls.original_sstate = bb_vars['SSTATE_DIR']
>     cls.devtool_sstate = os.path.join(bb_vars['TOPDIR'], 'sstate_devtool')
>     cls.sstate_conf  = 'SSTATE_DIR = "%s"\n' % cls.devtool_sstate
>     cls.sstate_conf += ('SSTATE_MIRRORS += "file://.* file:///%s/PATH"\n'
>                           % cls.original_sstate)
> 
>     And remove it after the test run:
> 
>     def tearDownClass(cls):
>     ...
>     runCmd('rm -rf %s' % cls.devtool_sstate)
> 
>     Even a keep builddir doesn't help to keep the private sstate cache.
> 
>      > The autobuilder uses a single writeable cache for all builds and
>      > builders, mounted as read-write NFS partition.
> 
>     This doesn't help in this case. The devtool test clean parts of the
>     sstate cache and therefore use its private sstate cache and the
>     original
>     sstate cache as mirror. This is a problem if a test triggers a build of
>     dependencies.
> 
>     Regards
>         Stefan

Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback

[Richard Purdie]

On Wed, 2021-12-08 at 08:45 +0100, Stefan Herbrechtsmeier wrote:
> Hi Richard,
> 
> Am 07.12.2021 um 16:45 schrieb Richard Purdie:
> > On Mon, 2021-12-06 at 09:04 +0100, Stefan Herbrechtsmeier wrote:
> > > From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
> > > 
> > > The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)'
> > > sets the branch= parameter too early to master and thereby breaks the
> > > -B/--srcbranch option.
> > > 
> > > ERROR: branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other
> > > 
> > > Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
> > > ---
> > > 
> > >   scripts/lib/recipetool/create.py | 15 +++++++--------
> > >   1 file changed, 7 insertions(+), 8 deletions(-)
> > 
> > I think something in this series is causing:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2926/steps/14/logs/stdio
> 
> Sorry I missed the adaption of the test test_devtool_add_fetch_git. It 
> doesn't expect a "branch=master" which is wrong.
> 
> - gitsm://git.yoctoproject.org/mraa;branch=master
> ?                                  --------------
> + gitsm://git.yoctoproject.org/mraa

I suspected something like that, thanks.

> I will update my patch series but need some time to test it locally. The 
> devtool test build a lot of recipe on every run because it use a private 
> sstate cache. How does the autobuilder overcome this problem?

The autobuilder doesn't. We stopped devtool using the public shared cache as it
is shared between autobuilder workers and one worker could delete artefacts
whilst another is using them. This caused all kinds of weird races and there
isn't much extra build time from the autobuilder perspective. The autobuilder
never deletes things from the shared cache outside of the timeout cron job which
removes objects that haven't been accessed in a month or so.

> > 
> > (the other selftest builds also failed)
> What do you mean by this?

I mean that we run oe-selftest on 4 different distros and they all failed the
same way so there were four failures in the build and I just pasted a link to
one of them.

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] recipetool: Set master branch only as fallback

[Richard Purdie]

On Wed, 2021-12-08 at 09:10 +0100, Alexander Kanavin wrote:
> On Wed, 8 Dec 2021 at 08:45, Stefan Herbrechtsmeier
> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
> > I will update my patch series but need some time to test it locally. The 
> > devtool test build a lot of recipe on every run because it use a private 
> > sstate cache. How does the autobuilder overcome this problem?
> > 
> 
> 
> If you don't rebase the branch used for testing, and set the cache to a
> directory outside of build directory (e.g. /home/stefan/sstate ) it will reuse
> that cache between selftest invocations.
> 
> The autobuilder uses a single writeable cache for all builds and builders,
> mounted as read-write NFS partition.

Stefan is correct here, it deliberately avoids the main cache. It does this as
it deletes things from it to run the test and we cannot do that on the public
cache.

Cheers,

Richard