[PATCH 1/3] tests: Remove redundant lfs=1 URI paramter

[PATCH 1/3] tests: Remove redundant lfs=1 URI paramter

[Paulo Neves]

lfs=1 is assumed unless explicitly disabled with lfs=0
This way we test the implicit case, which is also the
default for most users.

Signed-off-by: Paulo Neves <paulo@myneves.com>
---
 lib/bb/tests/fetch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index f3890321d..122d7e05f 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -2235,7 +2235,7 @@ class GitLfsTest(FetcherTest):
     def test_lfs_enabled(self):
         import shutil

-        uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir
+        uri = 'git://%s;protocol=file;branch=master' % self.srcdir
         self.d.setVar('SRC_URI', uri)

         # Careful: suppress initial attempt at downloading until
--
2.34.1

[PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs

[Paulo Neves]

Due to the datastore not having the PATH variable set, _find_git_lfs
would never return true and thus fetcher's git lfs test logic was
unreachable.

Now we set PATH on the datastore of the test and if the test host has
git-lfs the test will actually test git-lfs logic.

Signed-off-by: Paulo Neves <paulo@myneves.com>
---
 lib/bb/tests/fetch.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 122d7e05f..a9ad247d9 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -2237,6 +2237,7 @@ class GitLfsTest(FetcherTest):

         uri = 'git://%s;protocol=file;branch=master' % self.srcdir
         self.d.setVar('SRC_URI', uri)
+        self.d.setVar('PATH', os.environ.get('PATH'))

         # Careful: suppress initial attempt at downloading until
         # we know whether git-lfs is installed.
--
2.34.1

[PATCH 3/3] tests: Make sure test_lfs_enabled is ran with git lfs available

[Paulo Neves]

If git lfs does not exist in the host the test does not run all
the code paths possible, thus is not reproducile

Signed-off-by: Paulo Neves <paulo@myneves.com>
---
 lib/bb/tests/fetch.py | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index a9ad247d9..9b18f159d 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -2243,13 +2243,12 @@ class GitLfsTest(FetcherTest):
         # we know whether git-lfs is installed.
         fetcher, ud = self.fetch(uri=None, download=False)
         self.assertIsNotNone(ud.method._find_git_lfs)
+        self.assertTrue(ud.method._find_git_lfs(self.d),
+                msg="git-lfs not found. Cannot test git-lfs without git-lfs in the PATH")

-        # If git-lfs can be found, the unpack should be successful. Only
-        # attempt this with the real live copy of git-lfs installed.
-        if ud.method._find_git_lfs(self.d):
-            fetcher.download()
-            shutil.rmtree(self.gitdir, ignore_errors=True)
-            fetcher.unpack(self.d.getVar('WORKDIR'))
+        fetcher.download()
+        shutil.rmtree(self.gitdir, ignore_errors=True)
+        fetcher.unpack(self.d.getVar('WORKDIR'))

         # If git-lfs cannot be found, the unpack should throw an error
         with self.assertRaises(bb.fetch2.FetchError):
--
2.34.1

Re: [bitbake-devel] [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs

[Alexandre Belloni]

Hello,

This fails on the AB with:

FAIL: test_lfs_enabled (bb.tests.fetch.GitLfsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/bitbake/lib/bb/tests/fetch.py", line 2246, in test_lfs_enabled
    self.assertTrue(ud.method._find_git_lfs(self.d),
AssertionError: False is not true : git-lfs not found. Cannot test git-lfs without git-lfs in the PATH


On 10/02/2023 13:39:32+0000, Paulo Neves wrote:
> Due to the datastore not having the PATH variable set, _find_git_lfs
> would never return true and thus fetcher's git lfs test logic was
> unreachable.
> 
> Now we set PATH on the datastore of the test and if the test host has
> git-lfs the test will actually test git-lfs logic.
> 
> Signed-off-by: Paulo Neves <paulo@myneves.com>
> ---
>  lib/bb/tests/fetch.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
> index 122d7e05f..a9ad247d9 100644
> --- a/lib/bb/tests/fetch.py
> +++ b/lib/bb/tests/fetch.py
> @@ -2237,6 +2237,7 @@ class GitLfsTest(FetcherTest):
> 
>          uri = 'git://%s;protocol=file;branch=master' % self.srcdir
>          self.d.setVar('SRC_URI', uri)
> +        self.d.setVar('PATH', os.environ.get('PATH'))
> 
>          # Careful: suppress initial attempt at downloading until
>          # we know whether git-lfs is installed.
> --
> 2.34.1
> 
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14392): https://lists.openembedded.org/g/bitbake-devel/message/14392
> Mute This Topic: https://lists.openembedded.org/mt/96875983/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [bitbake-devel] [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs

[Paulo Neves]

This means that the test cannot run in the autobuilder and should
probably be removed or re-done.

My whole patchset needs to be re-done as well. The reason is that the
current git-lfs support in bitbake has very confusing behavior and very
host configuration dependent. This patch highlights that, as before it
we were supposedly testing git-lfs without git-lfs on the host. I am of
the opinion we should not do that.

There is no documentation anywhere on the behavior of the lfs parameter.
My current understanding is the following:

What the lfs=0/1 parameter does is to force the filter to run on the
DL_DIR's bare repository by doing a checkout to a temporary directory.
If lfs is unset, bitbake will still checkout lfs contents but will do it
during the unpack method which usually happens in the WORKDIR. git does
this transparently on git checkout. This is not ideal as it means that
there can be network access during the do_unpack task. This situation is
only (possibly) avoided if the user sets lfs=1. Otherwise, the user will
have network access on the do_unpack task.

Expect a V2

Paulo Neves

On 2/14/23 00:37, Alexandre Belloni wrote:
> Hello,
>
> This fails on the AB with:
>
> FAIL: test_lfs_enabled (bb.tests.fetch.GitLfsTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/bitbake/lib/bb/tests/fetch.py", line 2246, in test_lfs_enabled
>      self.assertTrue(ud.method._find_git_lfs(self.d),
> AssertionError: False is not true : git-lfs not found. Cannot test git-lfs without git-lfs in the PATH
>
>
> On 10/02/2023 13:39:32+0000, Paulo Neves wrote:
>> Due to the datastore not having the PATH variable set, _find_git_lfs
>> would never return true and thus fetcher's git lfs test logic was
>> unreachable.
>>
>> Now we set PATH on the datastore of the test and if the test host has
>> git-lfs the test will actually test git-lfs logic.
>>
>> Signed-off-by: Paulo Neves <paulo@myneves.com>
>> ---
>>   lib/bb/tests/fetch.py | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
>> index 122d7e05f..a9ad247d9 100644
>> --- a/lib/bb/tests/fetch.py
>> +++ b/lib/bb/tests/fetch.py
>> @@ -2237,6 +2237,7 @@ class GitLfsTest(FetcherTest):
>>
>>           uri = 'git://%s;protocol=file;branch=master' % self.srcdir
>>           self.d.setVar('SRC_URI', uri)
>> +        self.d.setVar('PATH', os.environ.get('PATH'))
>>
>>           # Careful: suppress initial attempt at downloading until
>>           # we know whether git-lfs is installed.
>> --
>> 2.34.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#14392): https://lists.openembedded.org/g/bitbake-devel/message/14392
>> Mute This Topic: https://lists.openembedded.org/mt/96875983/3617179
>> Group Owner: bitbake-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com