[PATCH 0/2] git/gitsm do_clean fixes

[PATCH 0/2] git/gitsm do_clean fixes

[nicomueller1991]

This pull request adds two fixes to the fetch2 git and gitsm modules.
Previously `do_cleanall` did not correctly remove all tarballs when
different combinations of BB_GENERATE_MIRROR_TARBALLS BB_GENERATE_SHALLOW_TARBALLS
and gitsm have been used.

The following changes since commit aec83663aadc871354b441eef0a3a41eb3780b13:

  build-appliance-image: Update to dunfell head revision (2023-02-13 07:48:24 +0000)

are available in the Git repository at:

  https://git.yoctoproject.org/poky-contrib niecore/fix-clean-tasks-for-git-and-gitsm
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=niecore/fix-clean-tasks-for-git-and-gitsm

Nico Müller (2):
  bitbake: fetch2/git: Clean all mirrortarballs
  bitbake: fetch2/gitsm: Add clean function

 bitbake/lib/bb/fetch2/git.py   |  6 +++++-
 bitbake/lib/bb/fetch2/gitsm.py | 19 +++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)

-- 
2.39.0

[PATCH 1/2] bitbake: fetch2/git: Clean all mirrortarballs

["Nico Müller]

From: Nico Müller <nico.mueller@de.bosch.com>

When bitbake is started with BB_GERNERATE_SHALLOW_TARBALLS,
this tarballs are not deleted during the clean step.
With this addition all tarballs are removed.

Signed-off-by: Nico Müller <nico.mueller@de.bosch.com>
---
 bitbake/lib/bb/fetch2/git.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index cad1ae8207b..9446ab01af8 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -591,8 +591,12 @@ class Git(FetchMethod):
 
     def clean(self, ud, d):
         """ clean the git directory """
+        to_remove = [ud.localpath]
+        for mirrortarball in ud.mirrortarballs:
+            fullmirrortarball = os.path.join(d.getVar("DL_DIR"), mirrortarball)
+            to_remove.append(fullmirrortarball)
+            to_remove.append(fullmirrortarball + ".done")
 
-        to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror + ".done"]
         # The localpath is a symlink to clonedir when it is cloned from a
         # mirror, so remove both of them.
         if os.path.islink(ud.localpath):
-- 
2.39.0

[PATCH 2/2] bitbake: fetch2/gitsm: Add clean function

["Nico Müller]

From: Nico Müller <nico.mueller@de.bosch.com>

The gitsm module previously did not clean any of the
tarballs created by the submodule since only the main
repo has been cleaned. With this addition firstly
all submodules are cleaned fully and after that
the main repository.

Signed-off-by: Nico Müller <nico.mueller@de.bosch.com>
---
 bitbake/lib/bb/fetch2/gitsm.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
index e7083001dae..5856e97f16d 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -182,6 +182,25 @@ class GitSM(Git):
         else:
             self.process_submodules(ud, ud.clonedir, download_submodule, d)
 
+    def clean(self, ud, d):
+        def clean_submodule(ud, url, module, modpath, workdir, d):
+            url += ";bareclone=1;nobranch=1"
+            try:
+                newclean = Fetch([url], d, cache=False)
+                newclean.clean()
+            except Exception as e:
+                logger.error('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e)))
+        
+        if ud.shallow and os.path.exists(ud.fullshallow):
+            tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR"))
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir)
+            self.process_submodules(ud, tmpdir, clean_submodule, d)
+            shutil.rmtree(tmpdir)
+        else:
+            self.process_submodules(ud, ud.clonedir, clean_submodule, d)        
+
+        Git.clean(self, ud, d)
+
     def unpack(self, ud, destdir, d):
         def unpack_submodules(ud, url, module, modpath, workdir, d):
             url += ";bareclone=1;nobranch=1"
-- 
2.39.0

Re: [poky] [PATCH 0/2] git/gitsm do_clean fixes

[Alexander Kanavin]

The patches need to go to bitbake-devel.

Also, are they for dunfell? Have they been tested on top of master,
are they maybe no longer needed?

Alex

On Fri, 24 Mar 2023 at 11:26, "Nico Müller <nicomueller1991@gmail.com> wrote:
>
> This pull request adds two fixes to the fetch2 git and gitsm modules.
> Previously `do_cleanall` did not correctly remove all tarballs when
> different combinations of BB_GENERATE_MIRROR_TARBALLS BB_GENERATE_SHALLOW_TARBALLS
> and gitsm have been used.
>
> The following changes since commit aec83663aadc871354b441eef0a3a41eb3780b13:
>
>   build-appliance-image: Update to dunfell head revision (2023-02-13 07:48:24 +0000)
>
> are available in the Git repository at:
>
>   https://git.yoctoproject.org/poky-contrib niecore/fix-clean-tasks-for-git-and-gitsm
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=niecore/fix-clean-tasks-for-git-and-gitsm
>
> Nico Müller (2):
>   bitbake: fetch2/git: Clean all mirrortarballs
>   bitbake: fetch2/gitsm: Add clean function
>
>  bitbake/lib/bb/fetch2/git.py   |  6 +++++-
>  bitbake/lib/bb/fetch2/gitsm.py | 19 +++++++++++++++++++
>  2 files changed, 24 insertions(+), 1 deletion(-)
>
> --
> 2.39.0
>
>
> 
>

Re: [PATCH 0/2] git/gitsm do_clean fixes

["Nico Müller]

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

Okay Alex, I will resubmit the patches to `bitbake-devel`.

The patches are created on dunfell. I checked the relevant master source files and the issue definitely persists there too.
Would you suggest to rebase the patches before submitting them again?

Best Regards,
Nico

[-- Attachment #2: Type: text/html, Size: 305 bytes --]

Re: [poky] [PATCH 0/2] git/gitsm do_clean fixes

[Alexander Kanavin]

Yes, there is a strict 'master first' policy to avoid losing patches
from old branches and general chaos.

I would suggest this sequence:
- set up a master build, verify that the issue still exists
- rebase the patches into it, verify that the issue no longer exists
- adjust the commit messages, showing how the issue can be observed,
and what the patch does so that it no longer happens
- submit to bitbake-devel

Alex

On Fri, 24 Mar 2023 at 12:17, "Nico Müller <nicomueller1991@gmail.com> wrote:
>
> Okay Alex, I will resubmit the patches to `bitbake-devel`.
>
> The patches are created on dunfell. I checked the relevant master source files and the issue definitely persists there too.
> Would you suggest to rebase the patches before submitting them again?
>
> Best Regards,
> Nico
> 
>