bitbake-devel.lists.openembedded.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 1/3] tests: Remove redundant lfs=1 URI paramter
@ 2023-02-10 13:39 Paulo Neves
  2023-02-10 13:39 ` [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs Paulo Neves
  2023-02-10 13:39 ` [PATCH 3/3] tests: Make sure test_lfs_enabled is ran with git lfs available Paulo Neves
  0 siblings, 2 replies; 5+ messages in thread
From: Paulo Neves @ 2023-02-10 13:39 UTC (permalink / raw)
  To: bitbake-devel; +Cc: Paulo Neves

lfs=1 is assumed unless explicitly disabled with lfs=0
This way we test the implicit case, which is also the
default for most users.

Signed-off-by: Paulo Neves <paulo@myneves.com>
---
 lib/bb/tests/fetch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index f3890321d..122d7e05f 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -2235,7 +2235,7 @@ class GitLfsTest(FetcherTest):
     def test_lfs_enabled(self):
         import shutil

-        uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir
+        uri = 'git://%s;protocol=file;branch=master' % self.srcdir
         self.d.setVar('SRC_URI', uri)

         # Careful: suppress initial attempt at downloading until
--
2.34.1




^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs
  2023-02-10 13:39 [PATCH 1/3] tests: Remove redundant lfs=1 URI paramter Paulo Neves
@ 2023-02-10 13:39 ` Paulo Neves
  2023-02-13 23:37   ` [bitbake-devel] " Alexandre Belloni
  2023-02-10 13:39 ` [PATCH 3/3] tests: Make sure test_lfs_enabled is ran with git lfs available Paulo Neves
  1 sibling, 1 reply; 5+ messages in thread
From: Paulo Neves @ 2023-02-10 13:39 UTC (permalink / raw)
  To: bitbake-devel; +Cc: Paulo Neves

Due to the datastore not having the PATH variable set, _find_git_lfs
would never return true and thus fetcher's git lfs test logic was
unreachable.

Now we set PATH on the datastore of the test and if the test host has
git-lfs the test will actually test git-lfs logic.

Signed-off-by: Paulo Neves <paulo@myneves.com>
---
 lib/bb/tests/fetch.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 122d7e05f..a9ad247d9 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -2237,6 +2237,7 @@ class GitLfsTest(FetcherTest):

         uri = 'git://%s;protocol=file;branch=master' % self.srcdir
         self.d.setVar('SRC_URI', uri)
+        self.d.setVar('PATH', os.environ.get('PATH'))

         # Careful: suppress initial attempt at downloading until
         # we know whether git-lfs is installed.
--
2.34.1




^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 3/3] tests: Make sure test_lfs_enabled is ran with git lfs available
  2023-02-10 13:39 [PATCH 1/3] tests: Remove redundant lfs=1 URI paramter Paulo Neves
  2023-02-10 13:39 ` [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs Paulo Neves
@ 2023-02-10 13:39 ` Paulo Neves
  1 sibling, 0 replies; 5+ messages in thread
From: Paulo Neves @ 2023-02-10 13:39 UTC (permalink / raw)
  To: bitbake-devel; +Cc: Paulo Neves

If git lfs does not exist in the host the test does not run all
the code paths possible, thus is not reproducile

Signed-off-by: Paulo Neves <paulo@myneves.com>
---
 lib/bb/tests/fetch.py | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index a9ad247d9..9b18f159d 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -2243,13 +2243,12 @@ class GitLfsTest(FetcherTest):
         # we know whether git-lfs is installed.
         fetcher, ud = self.fetch(uri=None, download=False)
         self.assertIsNotNone(ud.method._find_git_lfs)
+        self.assertTrue(ud.method._find_git_lfs(self.d),
+                msg="git-lfs not found. Cannot test git-lfs without git-lfs in the PATH")

-        # If git-lfs can be found, the unpack should be successful. Only
-        # attempt this with the real live copy of git-lfs installed.
-        if ud.method._find_git_lfs(self.d):
-            fetcher.download()
-            shutil.rmtree(self.gitdir, ignore_errors=True)
-            fetcher.unpack(self.d.getVar('WORKDIR'))
+        fetcher.download()
+        shutil.rmtree(self.gitdir, ignore_errors=True)
+        fetcher.unpack(self.d.getVar('WORKDIR'))

         # If git-lfs cannot be found, the unpack should throw an error
         with self.assertRaises(bb.fetch2.FetchError):
--
2.34.1




^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs
  2023-02-10 13:39 ` [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs Paulo Neves
@ 2023-02-13 23:37   ` Alexandre Belloni
  2023-02-14 11:26     ` Paulo Neves
  0 siblings, 1 reply; 5+ messages in thread
From: Alexandre Belloni @ 2023-02-13 23:37 UTC (permalink / raw)
  To: Paulo Neves; +Cc: bitbake-devel

Hello,

This fails on the AB with:

FAIL: test_lfs_enabled (bb.tests.fetch.GitLfsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/bitbake/lib/bb/tests/fetch.py", line 2246, in test_lfs_enabled
    self.assertTrue(ud.method._find_git_lfs(self.d),
AssertionError: False is not true : git-lfs not found. Cannot test git-lfs without git-lfs in the PATH


On 10/02/2023 13:39:32+0000, Paulo Neves wrote:
> Due to the datastore not having the PATH variable set, _find_git_lfs
> would never return true and thus fetcher's git lfs test logic was
> unreachable.
> 
> Now we set PATH on the datastore of the test and if the test host has
> git-lfs the test will actually test git-lfs logic.
> 
> Signed-off-by: Paulo Neves <paulo@myneves.com>
> ---
>  lib/bb/tests/fetch.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
> index 122d7e05f..a9ad247d9 100644
> --- a/lib/bb/tests/fetch.py
> +++ b/lib/bb/tests/fetch.py
> @@ -2237,6 +2237,7 @@ class GitLfsTest(FetcherTest):
> 
>          uri = 'git://%s;protocol=file;branch=master' % self.srcdir
>          self.d.setVar('SRC_URI', uri)
> +        self.d.setVar('PATH', os.environ.get('PATH'))
> 
>          # Careful: suppress initial attempt at downloading until
>          # we know whether git-lfs is installed.
> --
> 2.34.1
> 
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14392): https://lists.openembedded.org/g/bitbake-devel/message/14392
> Mute This Topic: https://lists.openembedded.org/mt/96875983/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs
  2023-02-13 23:37   ` [bitbake-devel] " Alexandre Belloni
@ 2023-02-14 11:26     ` Paulo Neves
  0 siblings, 0 replies; 5+ messages in thread
From: Paulo Neves @ 2023-02-14 11:26 UTC (permalink / raw)
  To: Alexandre Belloni; +Cc: bitbake-devel

This means that the test cannot run in the autobuilder and should
probably be removed or re-done.

My whole patchset needs to be re-done as well. The reason is that the
current git-lfs support in bitbake has very confusing behavior and very
host configuration dependent. This patch highlights that, as before it
we were supposedly testing git-lfs without git-lfs on the host. I am of
the opinion we should not do that.

There is no documentation anywhere on the behavior of the lfs parameter.
My current understanding is the following:

What the lfs=0/1 parameter does is to force the filter to run on the
DL_DIR's bare repository by doing a checkout to a temporary directory.
If lfs is unset, bitbake will still checkout lfs contents but will do it
during the unpack method which usually happens in the WORKDIR. git does
this transparently on git checkout. This is not ideal as it means that
there can be network access during the do_unpack task. This situation is
only (possibly) avoided if the user sets lfs=1. Otherwise, the user will
have network access on the do_unpack task.

Expect a V2

Paulo Neves

On 2/14/23 00:37, Alexandre Belloni wrote:
> Hello,
>
> This fails on the AB with:
>
> FAIL: test_lfs_enabled (bb.tests.fetch.GitLfsTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/bitbake/lib/bb/tests/fetch.py", line 2246, in test_lfs_enabled
>      self.assertTrue(ud.method._find_git_lfs(self.d),
> AssertionError: False is not true : git-lfs not found. Cannot test git-lfs without git-lfs in the PATH
>
>
> On 10/02/2023 13:39:32+0000, Paulo Neves wrote:
>> Due to the datastore not having the PATH variable set, _find_git_lfs
>> would never return true and thus fetcher's git lfs test logic was
>> unreachable.
>>
>> Now we set PATH on the datastore of the test and if the test host has
>> git-lfs the test will actually test git-lfs logic.
>>
>> Signed-off-by: Paulo Neves <paulo@myneves.com>
>> ---
>>   lib/bb/tests/fetch.py | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
>> index 122d7e05f..a9ad247d9 100644
>> --- a/lib/bb/tests/fetch.py
>> +++ b/lib/bb/tests/fetch.py
>> @@ -2237,6 +2237,7 @@ class GitLfsTest(FetcherTest):
>>
>>           uri = 'git://%s;protocol=file;branch=master' % self.srcdir
>>           self.d.setVar('SRC_URI', uri)
>> +        self.d.setVar('PATH', os.environ.get('PATH'))
>>
>>           # Careful: suppress initial attempt at downloading until
>>           # we know whether git-lfs is installed.
>> --
>> 2.34.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#14392): https://lists.openembedded.org/g/bitbake-devel/message/14392
>> Mute This Topic: https://lists.openembedded.org/mt/96875983/3617179
>> Group Owner: bitbake-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com




^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2023-02-14 11:27 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-10 13:39 [PATCH 1/3] tests: Remove redundant lfs=1 URI paramter Paulo Neves
2023-02-10 13:39 ` [PATCH 2/3] tests: Make test_lfs_enabled actually test git-lfs Paulo Neves
2023-02-13 23:37   ` [bitbake-devel] " Alexandre Belloni
2023-02-14 11:26     ` Paulo Neves
2023-02-10 13:39 ` [PATCH 3/3] tests: Make sure test_lfs_enabled is ran with git lfs available Paulo Neves

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).