All of lore.kernel.org
 help / color / mirror / Atom feed
* [1.50][PATCH 0/8] Review request
@ 2021-11-16  3:32 Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 1/8] bitbake:toaster:test: Update SSTATE URL Anuj Mittal
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

Please review these changes for 1.50. Tested with oe-core and no issues
seen on autobuilder.

Thanks,

Anuj

The following changes since commit bf59c0080763e8d03f75fe130ae9708e572391f3:

  fetch/wget: Add timeout for checkstatus calls (30s) (2021-11-11 10:59:06 +0000)

are available in the Git repository at:

  git://push.openembedded.org/bitbake-contrib stable/1.50-next

Jon Mason (1):
  bitbake:toaster:test: Update SSTATE URL

Jose Quaresma (1):
  cooker: check if upstream hash equivalence server is available

Richard Purdie (6):
  fetch2/perforce: Fix typo
  tests/fetch: Update github urls
  fetch: Handle mirror user/password replacements correctly
  tests/fetch: Update pcre.org address after github changes
  runqueue: Fix runall option task deletion ordering issue
  runqueue: Fix runall option handling

 lib/bb/cooker.py                      | 12 +++++++++-
 lib/bb/fetch2/__init__.py             |  4 ++++
 lib/bb/fetch2/perforce.py             |  2 +-
 lib/bb/runqueue.py                    | 34 +++++++++++++--------------
 lib/bb/tests/fetch.py                 | 32 ++++++++++++++-----------
 lib/toaster/tests/builds/buildtest.py |  2 +-
 6 files changed, 51 insertions(+), 35 deletions(-)

-- 
2.33.1



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [1.50][PATCH 1/8] bitbake:toaster:test: Update SSTATE URL
  2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
@ 2021-11-16  3:32 ` Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 2/8] fetch2/perforce: Fix typo Anuj Mittal
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

From: Jon Mason <jdmason@kudzu.us>

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b3c0dbddd7eb3c87e3989977d7640f09b49a460b)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/toaster/tests/builds/buildtest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/toaster/tests/builds/buildtest.py b/lib/toaster/tests/builds/buildtest.py
index 872bbd37..13b51fb0 100644
--- a/lib/toaster/tests/builds/buildtest.py
+++ b/lib/toaster/tests/builds/buildtest.py
@@ -119,7 +119,7 @@ class BuildTest(unittest.TestCase):
         if os.environ.get("TOASTER_TEST_USE_SSTATE_MIRROR"):
             ProjectVariable.objects.get_or_create(
                 name="SSTATE_MIRRORS",
-                value="file://.* http://autobuilder.yoctoproject.org/pub/sstate/PATH;downloadfilename=PATH",
+                value="file://.* http://sstate.yoctoproject.org/PATH;downloadfilename=PATH",
                 project=project)
 
         ProjectTarget.objects.create(project=project,
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [1.50][PATCH 2/8] fetch2/perforce: Fix typo
  2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 1/8] bitbake:toaster:test: Update SSTATE URL Anuj Mittal
@ 2021-11-16  3:32 ` Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 3/8] tests/fetch: Update github urls Anuj Mittal
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cf421235bd6f5eb12e9193634c0e870ab035b191)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/fetch2/perforce.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/perforce.py b/lib/bb/fetch2/perforce.py
index e2a41a4a..3b6fa4b1 100644
--- a/lib/bb/fetch2/perforce.py
+++ b/lib/bb/fetch2/perforce.py
@@ -134,7 +134,7 @@ class Perforce(FetchMethod):
 
         ud.setup_revisions(d)
 
-        ud.localfile = d.expand('%s_%s_%s_%s.tar.gz' % (cleanedhost, cleanedpath, cleandedmodule, ud.revision))
+        ud.localfile = d.expand('%s_%s_%s_%s.tar.gz' % (cleanedhost, cleanedpath, cleanedmodule, ud.revision))
 
     def _buildp4command(self, ud, d, command, depot_filename=None):
         """
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [1.50][PATCH 3/8] tests/fetch: Update github urls
  2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 1/8] bitbake:toaster:test: Update SSTATE URL Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 2/8] fetch2/perforce: Fix typo Anuj Mittal
@ 2021-11-16  3:32 ` Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 4/8] fetch: Handle mirror user/password replacements correctly Anuj Mittal
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 07fca7e3ab696ba985b3ef86ab9031d688bf2df2)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/tests/fetch.py | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index f5d557e8..c3e714ae 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -491,7 +491,7 @@ class GitDownloadDirectoryNamingTest(FetcherTest):
         super(GitDownloadDirectoryNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake"
         self.recipe_dir = "git.openembedded.org.bitbake"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git"
+        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https"
         self.mirror_dir = "github.com.openembedded.bitbake.git"
 
         self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
@@ -539,7 +539,7 @@ class TarballNamingTest(FetcherTest):
         super(TarballNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake"
         self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git"
+        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https"
         self.mirror_tarball = "git2_github.com.openembedded.bitbake.git.tar.gz"
 
         self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1')
@@ -573,7 +573,7 @@ class GitShallowTarballNamingTest(FetcherTest):
         super(GitShallowTarballNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake"
         self.recipe_tarball = "gitshallow_git.openembedded.org.bitbake_82ea737-1_master.tar.gz"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git"
+        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https"
         self.mirror_tarball = "gitshallow_github.com.openembedded.bitbake.git_82ea737-1_master.tar.gz"
 
         self.d.setVar('BB_GIT_SHALLOW', '1')
@@ -985,7 +985,7 @@ class FetcherNetworkTest(FetcherTest):
     def test_git_submodule_dbus_broker(self):
         # The following external repositories have show failures in fetch and unpack operations
         # We want to avoid regressions!
-        url = "gitsm://github.com/bus1/dbus-broker;protocol=git;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main"
+        url = "gitsm://github.com/bus1/dbus-broker;protocol=https;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main"
         fetcher = bb.fetch.Fetch([url], self.d)
         fetcher.download()
         # Previous cwd has been deleted
@@ -1001,7 +1001,7 @@ class FetcherNetworkTest(FetcherTest):
 
     @skipIfNoNetwork()
     def test_git_submodule_CLI11(self):
-        url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main"
+        url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main"
         fetcher = bb.fetch.Fetch([url], self.d)
         fetcher.download()
         # Previous cwd has been deleted
@@ -1016,12 +1016,12 @@ class FetcherNetworkTest(FetcherTest):
     @skipIfNoNetwork()
     def test_git_submodule_update_CLI11(self):
         """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """
-        url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=main"
+        url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=main"
         fetcher = bb.fetch.Fetch([url], self.d)
         fetcher.download()
 
         # CLI11 that pulls in a newer nlohmann-json
-        url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=main"
+        url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=main"
         fetcher = bb.fetch.Fetch([url], self.d)
         fetcher.download()
         # Previous cwd has been deleted
@@ -1035,7 +1035,7 @@ class FetcherNetworkTest(FetcherTest):
 
     @skipIfNoNetwork()
     def test_git_submodule_aktualizr(self):
-        url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
+        url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=https;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
         fetcher = bb.fetch.Fetch([url], self.d)
         fetcher.download()
         # Previous cwd has been deleted
@@ -1055,7 +1055,7 @@ class FetcherNetworkTest(FetcherTest):
         """ Prevent regression on deeply nested submodules not being checked out properly, even though they were fetched. """
 
         # This repository also has submodules where the module (name), path and url do not align
-        url = "gitsm://github.com/azure/iotedge.git;protocol=git;rev=d76e0316c6f324345d77c48a83ce836d09392699"
+        url = "gitsm://github.com/azure/iotedge.git;protocol=https;rev=d76e0316c6f324345d77c48a83ce836d09392699"
         fetcher = bb.fetch.Fetch([url], self.d)
         fetcher.download()
         # Previous cwd has been deleted
@@ -1231,7 +1231,7 @@ class FetchLatestVersionTest(FetcherTest):
 
     test_git_uris = {
         # version pattern "X.Y.Z"
-        ("mx-1.0", "git://github.com/clutter-project/mx.git;branch=mx-1.4", "9b1db6b8060bd00b121a692f942404a24ae2960f", "")
+        ("mx-1.0", "git://github.com/clutter-project/mx.git;branch=mx-1.4;protocol=https", "9b1db6b8060bd00b121a692f942404a24ae2960f", "")
             : "1.99.4",
         # version pattern "vX.Y"
         # mirror of git.infradead.org since network issues interfered with testing
@@ -1258,9 +1258,9 @@ class FetchLatestVersionTest(FetcherTest):
             : "0.4.3",
         ("build-appliance-image", "git://git.yoctoproject.org/poky", "b37dd451a52622d5b570183a81583cc34c2ff555", "(?P<pver>(([0-9][\.|_]?)+[0-9]))")
             : "11.0.0",
-        ("chkconfig-alternatives-native", "git://github.com/kergoth/chkconfig;branch=sysroot", "cd437ecbd8986c894442f8fce1e0061e20f04dee", "chkconfig\-(?P<pver>((\d+[\.\-_]*)+))")
+        ("chkconfig-alternatives-native", "git://github.com/kergoth/chkconfig;branch=sysroot;protocol=https", "cd437ecbd8986c894442f8fce1e0061e20f04dee", "chkconfig\-(?P<pver>((\d+[\.\-_]*)+))")
             : "1.3.59",
-        ("remake", "git://github.com/rocky/remake.git", "f05508e521987c8494c92d9c2871aec46307d51d", "(?P<pver>(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))")
+        ("remake", "git://github.com/rocky/remake.git;protocol=https", "f05508e521987c8494c92d9c2871aec46307d51d", "(?P<pver>(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))")
             : "3.82+dbg0.9",
     }
 
@@ -2047,7 +2047,7 @@ class GitShallowTest(FetcherTest):
 
     @skipIfNoNetwork()
     def test_bitbake(self):
-        self.git('remote add --mirror=fetch origin git://github.com/openembedded/bitbake', cwd=self.srcdir)
+        self.git('remote add --mirror=fetch origin https://github.com/openembedded/bitbake', cwd=self.srcdir)
         self.git('config core.bare true', cwd=self.srcdir)
         self.git('fetch', cwd=self.srcdir)
 
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [1.50][PATCH 4/8] fetch: Handle mirror user/password replacements correctly
  2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
                   ` (2 preceding siblings ...)
  2021-11-16  3:32 ` [1.50][PATCH 3/8] tests/fetch: Update github urls Anuj Mittal
@ 2021-11-16  3:32 ` Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 5/8] cooker: check if upstream hash equivalence server is available Anuj Mittal
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.

[YOCTO #13823]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 66ad58bb87e5158aced572be4f1d5726bc97fcce)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/fetch2/__init__.py | 4 ++++
 lib/bb/tests/fetch.py     | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index dbf8b50e..1005ec10 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -430,6 +430,7 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
     uri_replace_decoded = list(decodeurl(uri_replace))
     logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded))
     result_decoded = ['', '', '', '', '', {}]
+    # 0 - type, 1 - host, 2 - path, 3 - user,  4- pswd, 5 - params
     for loc, i in enumerate(uri_find_decoded):
         result_decoded[loc] = uri_decoded[loc]
         regexp = i
@@ -449,6 +450,9 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
                 for l in replacements:
                     uri_replace_decoded[loc][k] = uri_replace_decoded[loc][k].replace(l, replacements[l])
                 result_decoded[loc][k] = uri_replace_decoded[loc][k]
+        elif (loc == 3 or loc == 4) and uri_replace_decoded[loc]:
+            # User/password in the replacement is just a straight replacement
+            result_decoded[loc] = uri_replace_decoded[loc]
         elif (re.match(regexp, uri_decoded[loc])):
             if not uri_replace_decoded[loc]:
                 result_decoded[loc] = ""
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index c3e714ae..0fb279e8 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -431,6 +431,10 @@ class MirrorUriTest(FetcherTest):
         ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", "git://someserver.org/bitbake;branch=master", "git://git.openembedded.org/bitbake;protocol=http")
             : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master;protocol=http",
 
+        ("git://user1@someserver.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", "git://someserver.org/bitbake;branch=master", "git://user2@git.openembedded.org/bitbake;protocol=http")
+            : "git://user2@git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master;protocol=http",
+
+
         #Renaming files doesn't work
         #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"
         #("file://sstate-xyz.tgz", "file://.*/.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [1.50][PATCH 5/8] cooker: check if upstream hash equivalence server is available
  2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
                   ` (3 preceding siblings ...)
  2021-11-16  3:32 ` [1.50][PATCH 4/8] fetch: Handle mirror user/password replacements correctly Anuj Mittal
@ 2021-11-16  3:32 ` Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 6/8] tests/fetch: Update pcre.org address after github changes Anuj Mittal
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

From: Jose Quaresma <quaresma.jose@gmail.com>

When the user specify an invalid upstream hash equivalence server in
BB_HASHSERVE_UPSTREAM notify the user that we can't connect the server.

Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit be45aeb9a84f30c28711e87e2d2a4a86320a8d94)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cooker.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 89f1fad0..b041d2a0 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -388,12 +388,22 @@ class BBCooker:
             # Create a new hash server bound to a unix domain socket
             if not self.hashserv:
                 dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db"
+                upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM") or None
+                if upstream:
+                    import socket
+                    try:
+                        sock = socket.create_connection(upstream.split(":"), 5)
+                        sock.close()
+                    except socket.error as e:
+                        bb.warn("BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at '%s': %s" 
+                                 % (upstream, repr(e)))
+
                 self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR")
                 self.hashserv = hashserv.create_server(
                     self.hashservaddr,
                     dbfile,
                     sync=False,
-                    upstream=self.data.getVar("BB_HASHSERVE_UPSTREAM") or None,
+                    upstream=upstream,
                 )
                 self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever)
                 self.hashserv.process.start()
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [1.50][PATCH 6/8] tests/fetch: Update pcre.org address after github changes
  2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
                   ` (4 preceding siblings ...)
  2021-11-16  3:32 ` [1.50][PATCH 5/8] cooker: check if upstream hash equivalence server is available Anuj Mittal
@ 2021-11-16  3:32 ` Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 7/8] runqueue: Fix runall option task deletion ordering issue Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 8/8] runqueue: Fix runall option handling Anuj Mittal
  7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

vcs.pcre.org was a redirect to github which we use for subversion testing.
With the protocol changes at github and the removal of the redirect, use a
direct address for github.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6230ca71eb7eb2a6db162e28a01727d00af5299b)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/tests/fetch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 0fb279e8..86eccf7d 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -1117,7 +1117,7 @@ class SVNTest(FetcherTest):
 
         bb.process.run("svn co %s svnfetch_co" % self.repo_url, cwd=self.tempdir)
         # Github will emulate SVN.  Use this to check if we're downloding...
-        bb.process.run("svn propset svn:externals 'bitbake svn://vcs.pcre.org/pcre2/code' .",
+        bb.process.run("svn propset svn:externals 'bitbake https://github.com/PhilipHazel/pcre2.git' .",
                        cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk'))
         bb.process.run("svn commit --non-interactive -m 'Add external'",
                        cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk'))
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [1.50][PATCH 7/8] runqueue: Fix runall option task deletion ordering issue
  2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
                   ` (5 preceding siblings ...)
  2021-11-16  3:32 ` [1.50][PATCH 6/8] tests/fetch: Update pcre.org address after github changes Anuj Mittal
@ 2021-11-16  3:32 ` Anuj Mittal
  2021-11-16  3:32 ` [1.50][PATCH 8/8] runqueue: Fix runall option handling Anuj Mittal
  7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The runbuild option handling in runqueue was flawed as items deleted from the
main task list may be dependencies and hence cause index errors.

Rather than modify runtaskentries straight away, compute a new shorted list
and use that as an input to the second phase. This avoids the need to add tasks
back to the list meaning delcount can be simplifed to a simple counter.

The second use case in runonly doen't re-add items so doesn't have this
issue.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3428e3c54eb5cc03ff96f9cee6dc839afee7a419)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py | 35 ++++++++++++++++-------------------
 1 file changed, 16 insertions(+), 19 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 10511a09..3300bb4f 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -926,38 +926,35 @@ class RunQueueData:
         #
         # Once all active tasks are marked, prune the ones we don't need.
 
-        delcount = {}
-        for tid in list(self.runtaskentries.keys()):
-            if tid not in runq_build:
-                delcount[tid] = self.runtaskentries[tid]
-                del self.runtaskentries[tid]
-
         # Handle --runall
         if self.cooker.configuration.runall:
             # re-run the mark_active and then drop unused tasks from new list
-            runq_build = {}
+            reduced_tasklist = set(self.runtaskentries.keys())
+            for tid in list(self.runtaskentries.keys()):
+                if tid not in runq_build:
+                   reduced_tasklist.remove(tid)
 
             for task in self.cooker.configuration.runall:
                 if not task.startswith("do_"):
                     task = "do_{0}".format(task)
                 runall_tids = set()
-                for tid in list(self.runtaskentries):
+                for tid in reduced_tasklist:
                     wanttid = "{0}:{1}".format(fn_from_tid(tid), task)
-                    if wanttid in delcount:
-                        self.runtaskentries[wanttid] = delcount[wanttid]
                     if wanttid in self.runtaskentries:
                         runall_tids.add(wanttid)
 
                 for tid in list(runall_tids):
-                    mark_active(tid,1)
+                    mark_active(tid, 1)
                     if self.cooker.configuration.force:
                         invalidate_task(tid, False)
 
-            for tid in list(self.runtaskentries.keys()):
-                if tid not in runq_build:
-                    delcount[tid] = self.runtaskentries[tid]
-                    del self.runtaskentries[tid]
+        delcount = set()
+        for tid in list(self.runtaskentries.keys()):
+            if tid not in runq_build:
+                delcount.add(tid)
+                del self.runtaskentries[tid]
 
+        if self.cooker.configuration.runall:
             if len(self.runtaskentries) == 0:
                 bb.msg.fatal("RunQueue", "Could not find any tasks with the tasknames %s to run within the recipes of the taskgraphs of the targets %s" % (str(self.cooker.configuration.runall), str(self.targets)))
 
@@ -971,16 +968,16 @@ class RunQueueData:
             for task in self.cooker.configuration.runonly:
                 if not task.startswith("do_"):
                     task = "do_{0}".format(task)
-                runonly_tids = { k: v for k, v in self.runtaskentries.items() if taskname_from_tid(k) == task }
+                runonly_tids = [k for k in self.runtaskentries.keys() if taskname_from_tid(k) == task]
 
-                for tid in list(runonly_tids):
-                    mark_active(tid,1)
+                for tid in runonly_tids:
+                    mark_active(tid, 1)
                     if self.cooker.configuration.force:
                         invalidate_task(tid, False)
 
             for tid in list(self.runtaskentries.keys()):
                 if tid not in runq_build:
-                    delcount[tid] = self.runtaskentries[tid]
+                    delcount.add(tid)
                     del self.runtaskentries[tid]
 
             if len(self.runtaskentries) == 0:
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [1.50][PATCH 8/8] runqueue: Fix runall option handling
  2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
                   ` (6 preceding siblings ...)
  2021-11-16  3:32 ` [1.50][PATCH 7/8] runqueue: Fix runall option task deletion ordering issue Anuj Mittal
@ 2021-11-16  3:32 ` Anuj Mittal
  7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-16  3:32 UTC (permalink / raw)
  To: bitbake-devel

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The previous fix for runall option handling had a small bug in it, it
didn't clear the originally processed task list which meant it was running
too many tasks. Fix this so the list is reset and rebuild correctly.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 87c9e120897ed04dfc64d4752fc602f9bfcb8645)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 3300bb4f..cd10da8b 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -933,6 +933,7 @@ class RunQueueData:
             for tid in list(self.runtaskentries.keys()):
                 if tid not in runq_build:
                    reduced_tasklist.remove(tid)
+            runq_build = {}
 
             for task in self.cooker.configuration.runall:
                 if not task.startswith("do_"):
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-11-16  3:32 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-16  3:32 [1.50][PATCH 0/8] Review request Anuj Mittal
2021-11-16  3:32 ` [1.50][PATCH 1/8] bitbake:toaster:test: Update SSTATE URL Anuj Mittal
2021-11-16  3:32 ` [1.50][PATCH 2/8] fetch2/perforce: Fix typo Anuj Mittal
2021-11-16  3:32 ` [1.50][PATCH 3/8] tests/fetch: Update github urls Anuj Mittal
2021-11-16  3:32 ` [1.50][PATCH 4/8] fetch: Handle mirror user/password replacements correctly Anuj Mittal
2021-11-16  3:32 ` [1.50][PATCH 5/8] cooker: check if upstream hash equivalence server is available Anuj Mittal
2021-11-16  3:32 ` [1.50][PATCH 6/8] tests/fetch: Update pcre.org address after github changes Anuj Mittal
2021-11-16  3:32 ` [1.50][PATCH 7/8] runqueue: Fix runall option task deletion ordering issue Anuj Mittal
2021-11-16  3:32 ` [1.50][PATCH 8/8] runqueue: Fix runall option handling Anuj Mittal

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.