bitbake-devel.lists.openembedded.org archive mirror
 help / color / mirror / Atom feed
* [bitbake][dunfell][1.46][PATCH 0/3] Patch review
@ 2023-02-07 15:09 Steve Sakoman
  2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 1/3] bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url Steve Sakoman
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Steve Sakoman @ 2023-02-07 15:09 UTC (permalink / raw)
  To: bitbake-devel

Please review this set of patches for dunfell/1.46 and have comments back by
end of day Thursday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4893

The following changes since commit c16d364dbf68d2a500fecaf8d6e6d62b11475d9f:

  bb/utils: include SSL certificate paths in export_proxies (2023-02-06 23:42:50 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Charlie Davies (1):
  bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url

Marek Vasut (2):
  fetch2/git: Prevent git fetcher from fetching gitlab repository
    metadata
  fetch2/git: Clarify the meaning of namespace

 .../bitbake-user-manual-fetching.rst          |  4 ++--
 lib/bb/fetch2/git.py                          | 20 ++++++++++++-------
 2 files changed, 15 insertions(+), 9 deletions(-)

-- 
2.25.1



^ permalink raw reply	[flat|nested] 5+ messages in thread

* [bitbake][dunfell][1.46][PATCH 1/3] bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url
  2023-02-07 15:09 [bitbake][dunfell][1.46][PATCH 0/3] Patch review Steve Sakoman
@ 2023-02-07 15:09 ` Steve Sakoman
  2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 2/3] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata Steve Sakoman
  2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 3/3] fetch2/git: Clarify the meaning of namespace Steve Sakoman
  2 siblings, 0 replies; 5+ messages in thread
From: Steve Sakoman @ 2023-02-07 15:09 UTC (permalink / raw)
  To: bitbake-devel

From: Charlie Davies <charles.davies@whitetree.xyz>

This commit replaces the instances where escaped double quotes
are used to support SRC_URI url containing spaces with the more
pythonic shlex.quote().

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4f9ba9c794de55bea0343267467bddea99844374)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/fetch2/git.py | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 63a9f92b..2868aa5d 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -63,6 +63,7 @@ import errno
 import fnmatch
 import os
 import re
+import shlex
 import subprocess
 import tempfile
 import bb
@@ -352,7 +353,7 @@ class Git(FetchMethod):
             # We do this since git will use a "-l" option automatically for local urls where possible
             if repourl.startswith("file://"):
                 repourl = repourl[7:]
-            clone_cmd = "LANG=C %s clone --bare --mirror \"%s\" %s --progress" % (ud.basecmd, repourl, ud.clonedir)
+            clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
@@ -364,8 +365,8 @@ class Git(FetchMethod):
             if "origin" in output:
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
 
-            runfetchcmd("%s remote add --mirror=fetch origin \"%s\"" % (ud.basecmd, repourl), d, workdir=ud.clonedir)
-            fetch_cmd = "LANG=C %s fetch -f --progress \"%s\" refs/*:refs/*" % (ud.basecmd, repourl)
+            runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
+            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
@@ -559,7 +560,7 @@ class Git(FetchMethod):
             raise bb.fetch2.UnpackError("No up to date source found: " + "; ".join(source_error), ud.url)
 
         repourl = self._get_repo_url(ud)
-        runfetchcmd("%s remote set-url origin \"%s\"" % (ud.basecmd, repourl), d, workdir=destdir)
+        runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=destdir)
 
         if self._contains_lfs(ud, d, destdir):
             if need_lfs and not self._find_git_lfs(d):
@@ -687,8 +688,8 @@ class Git(FetchMethod):
         d.setVar('_BB_GIT_IN_LSREMOTE', '1')
         try:
             repourl = self._get_repo_url(ud)
-            cmd = "%s ls-remote \"%s\" %s" % \
-                (ud.basecmd, repourl, search)
+            cmd = "%s ls-remote %s %s" % \
+                (ud.basecmd, shlex.quote(repourl), search)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, cmd, repourl)
             output = runfetchcmd(cmd, d, True)
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [bitbake][dunfell][1.46][PATCH 2/3] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata
  2023-02-07 15:09 [bitbake][dunfell][1.46][PATCH 0/3] Patch review Steve Sakoman
  2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 1/3] bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url Steve Sakoman
@ 2023-02-07 15:09 ` Steve Sakoman
  2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 3/3] fetch2/git: Clarify the meaning of namespace Steve Sakoman
  2 siblings, 0 replies; 5+ messages in thread
From: Steve Sakoman @ 2023-02-07 15:09 UTC (permalink / raw)
  To: bitbake-devel

From: Marek Vasut <marex@denx.de>

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every
single object in the remote repository. This works poorly with gitlab
and github, which use the remote git repository to track its metadata
like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/*
and refs/keep-around/* and they all contain massive amount of data that
are useless for the bitbake build purposes. The amount of useless data
can in fact be so massive (e.g. with FDO mesa.git repository) that some
proxies may outright terminate the 'git fetch' connection, and make it
appear as if bitbake got stuck on 'git fetch' with no output.

To avoid fetching all these useless metadata, tweak the git fetcher such
that it only fetches refs/heads/* and refs/tags/* . Avoid using negative
refspecs as those are only available in new git versions.

Per feedback on the ML, Gerrit may push commits outsides of branches or
tags during CI runs, which currently works with the 'nobranch=1' fetcher
parameter. To retain this functionality, keep fetching everything in case
the 'nobranch=1' is present. This still avoids fetching massive amount of
data in the common case, since 'nobranch=1' is rare. Update 'nobranch'
documentation.

Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | 4 ++--
 lib/bb/fetch2/git.py                                     | 8 ++++++--
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 93ac18b7..37c7bcc8 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -405,8 +405,8 @@ This fetcher supports the following parameters:
 
 -  *"nobranch":* Tells the fetcher to not check the SHA validation for
    the branch when set to "1". The default is "0". Set this option for
-   the recipe that refers to the commit that is valid for a tag instead
-   of the branch.
+   the recipe that refers to the commit that is valid for a any namespace
+   instead of the branch.
 
 -  *"bareclone":* Tells the fetcher to clone a bare clone into the
    destination directory without checking out a working tree. Only the
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 2868aa5d..b8e00ced 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -44,7 +44,7 @@ Supported SRC_URI options are:
 
 - nobranch
    Don't check the SHA validation for branch. set this option for the recipe
-   referring to commit which is valid in tag instead of branch.
+   referring to commit which is valid in any namespace instead of branch.
    The default is "0", set nobranch=1 if needed.
 
 - usehead
@@ -366,7 +366,11 @@ class Git(FetchMethod):
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
 
             runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
-            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
+
+            if ud.nobranch:
+                fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
+            else:
+                fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl))
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [bitbake][dunfell][1.46][PATCH 3/3] fetch2/git: Clarify the meaning of namespace
  2023-02-07 15:09 [bitbake][dunfell][1.46][PATCH 0/3] Patch review Steve Sakoman
  2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 1/3] bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url Steve Sakoman
  2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 2/3] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata Steve Sakoman
@ 2023-02-07 15:09 ` Steve Sakoman
  2 siblings, 0 replies; 5+ messages in thread
From: Steve Sakoman @ 2023-02-07 15:09 UTC (permalink / raw)
  To: bitbake-devel

From: Marek Vasut <marex@denx.de>

Namespace in this context means a branch, a tag, etc., clarify
it in the description. Also, fix a typo "a any", replace with
plain "any".

This patch is based of feedback on new applied patch
d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata")

Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | 4 ++--
 lib/bb/fetch2/git.py                                     | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 37c7bcc8..75e8dd69 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -405,8 +405,8 @@ This fetcher supports the following parameters:
 
 -  *"nobranch":* Tells the fetcher to not check the SHA validation for
    the branch when set to "1". The default is "0". Set this option for
-   the recipe that refers to the commit that is valid for a any namespace
-   instead of the branch.
+   the recipe that refers to the commit that is valid for any namespace
+   (branch, tag, ...) instead of the branch.
 
 -  *"bareclone":* Tells the fetcher to clone a bare clone into the
    destination directory without checking out a working tree. Only the
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index b8e00ced..cad1ae82 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -44,7 +44,8 @@ Supported SRC_URI options are:
 
 - nobranch
    Don't check the SHA validation for branch. set this option for the recipe
-   referring to commit which is valid in any namespace instead of branch.
+   referring to commit which is valid in any namespace (branch, tag, ...)
+   instead of branch.
    The default is "0", set nobranch=1 if needed.
 
 - usehead
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [bitbake][dunfell][1.46][PATCH 0/3] Patch review
@ 2021-11-28 22:02 Steve Sakoman
  0 siblings, 0 replies; 5+ messages in thread
From: Steve Sakoman @ 2021-11-28 22:02 UTC (permalink / raw)
  To: bitbake-devel

Please review this set of patches for dunfell/1.46 and have comments back by
end of day Tuesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2968

The following changes since commit c0348de8121c3a842bf44906f7e2f79e93f7275b:

  fetch/wget: Add timeout for checkstatus calls (30s) (2021-11-11 10:58:01 +0000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (3):
  command: Ensure exceptions inheriting from BBHandledException are
    visible
  tinfoil: When sending commands we need to process events
  process/knotty: Improve early exception handling

 lib/bb/command.py        |  6 +++++-
 lib/bb/server/process.py |  7 ++++++-
 lib/bb/tinfoil.py        | 11 ++++++++++-
 lib/bb/ui/knotty.py      | 21 +++++++++++++++++----
 4 files changed, 38 insertions(+), 7 deletions(-)

-- 
2.25.1



^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2023-02-07 15:10 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-07 15:09 [bitbake][dunfell][1.46][PATCH 0/3] Patch review Steve Sakoman
2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 1/3] bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url Steve Sakoman
2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 2/3] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata Steve Sakoman
2023-02-07 15:09 ` [bitbake][dunfell][1.46][PATCH 3/3] fetch2/git: Clarify the meaning of namespace Steve Sakoman
  -- strict thread matches above, loose matches on Subject: below --
2021-11-28 22:02 [bitbake][dunfell][1.46][PATCH 0/3] Patch review Steve Sakoman

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).