[PATCH v3 2/2] bitbake: fetch2/git: git-lfs check

[PATCH v3 2/2] bitbake: fetch2/git: git-lfs check

[Naveen Saini]

Build will fail if repository has lfs contents in absense of git-lfs tool on host.
Build will pass if repository may or may not contains lfs content if host has git-lfs installed.

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
---
 bitbake/lib/bb/fetch2/git.py | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 8185bf4..d862942 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -503,6 +503,15 @@ class Git(FetchMethod):
 
         repourl = self._get_repo_url(ud)
         runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d, workdir=destdir)
+
+        if self._contains_lfs(ud, d, destdir):
+            origbbenv = d.getVar("BB_ORIGENV", False)
+            path = origbbenv.getVar("PATH")
+            gitlfstool = bb.utils.which(path, "git-lfs", executable=True)
+            if not gitlfstool:
+                raise bb.fetch2.FetchError("Repository %s has lfs content, install git-lfs plugin on host to download" % (repourl))
+
+
         if not ud.nocheckout:
             if subdir != "":
                 runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d,
@@ -553,6 +562,21 @@ class Git(FetchMethod):
             raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
         return output.split()[0] != "0"
 
+    def _contains_lfs(self, ud, d, wd):
+        """
+        Check git lfs repository
+        """
+        cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
+                ud.basecmd)
+        try:
+            output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
+        except bb.fetch2.FetchError:
+            return False
+        if int(output) > 0:
+            return True
+        else:
+            return False
+
     def _get_repo_url(self, ud):
         """
         Return the repository URL
-- 
2.7.4

Re: [PATCH v3 2/2] bitbake: fetch2/git: git-lfs check

[Richard Purdie]

On Thu, 2019-04-25 at 09:54 +0800, Naveen Saini wrote:
> Build will fail if repository has lfs contents in absense of git-lfs tool on host.
> Build will pass if repository may or may not contains lfs content if host has git-lfs installed.

Bitbake patches need to go to the bitbake list.

> Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
> ---
>  bitbake/lib/bb/fetch2/git.py | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> index 8185bf4..d862942 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -503,6 +503,15 @@ class Git(FetchMethod):
>  
>          repourl = self._get_repo_url(ud)
>          runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d, workdir=destdir)
> +
> +        if self._contains_lfs(ud, d, destdir):
> +            origbbenv = d.getVar("BB_ORIGENV", False)
> +            path = origbbenv.getVar("PATH")
> +            gitlfstool = bb.utils.which(path, "git-lfs", executable=True)
> +            if not gitlfstool:
> +                raise bb.fetch2.FetchError("Repository %s has lfs content, install git-lfs plugin on host to download" % (repourl))

Can we just rely on the usual PATH here please, I don't think we need
to touch BB_ORIGENV? We can assume the environment will be setup with
git-lfs if its present.

> +
>          if not ud.nocheckout:
>              if subdir != "":
>                  runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d,
> @@ -553,6 +562,21 @@ class Git(FetchMethod):
>              raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
>          return output.split()[0] != "0"
>  
> +    def _contains_lfs(self, ud, d, wd):
> +        """
> +        Check git lfs repository

If we're going to add a comment, please make it useful, e.g.

Check if the repository has 'lfs' (large file) content

> +        """
> +        cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
> +                ud.basecmd)
> +        try:
> +            output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
> +        except bb.fetch2.FetchError:
> +            return False
> +        if int(output) > 0:
> +            return True
> +        else:
> +            return False

Can be slightly simplified to:

        try:
            output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
            if int(output) > 0:
               return True
        except
bb.fetch2.FetchError:
            pass
        return False

I did wonder if we could/should also catch an exception if the output
can't be turned into an int for some reason with:

        except bb.fetch2.FetchError, ValueError:

Cheers,

Richard