* git do_unpack
@ 2020-06-28 17:27 John Klug
2020-06-28 19:30 ` [OE-core] " Alexander Kanavin
0 siblings, 1 reply; 7+ messages in thread
From: John Klug @ 2020-06-28 17:27 UTC (permalink / raw)
To: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 739 bytes --]
Recently it has come to my attention that in morty, and maybe later, do_unpack for git does not use the downloads directory, and attempts to contact the git server.
This is unfortunate, as we often do bitbake -c fetchall and give tarballs of our workspace to customers, thinking it would save something, but it appears the savings are limited, and now I have a customer complaint about this, since I have one recipe that uses a private repository.
What is the logic of not using the downloads directory for the source of the unpack?
I recently fixed a recipe by writing an unpack that does not use networking and does a clone of the downloads directory to get around this. Why is this not done by openembedded-core?
Thanks.
[-- Attachment #2: Type: text/html, Size: 781 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] git do_unpack
2020-06-28 17:27 git do_unpack John Klug
@ 2020-06-28 19:30 ` Alexander Kanavin
2020-06-30 4:45 ` John Klug
0 siblings, 1 reply; 7+ messages in thread
From: Alexander Kanavin @ 2020-06-28 19:30 UTC (permalink / raw)
To: John Klug; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]
It would help if you can:
a) provide log.do_unpack that shows that network access is indeed happening;
b) demonstrate the behavior with oe-core master, as morty is soon eligible
for retro computing museum, and hardly anyone wants to look into that :)
Alex
On Sun, 28 Jun 2020 at 19:27, John Klug <john.klug@multitech.com> wrote:
> Recently it has come to my attention that in morty, and maybe later,
> do_unpack for git does not use the downloads directory, and attempts to
> contact the git server.
>
> This is unfortunate, as we often do bitbake -c fetchall and give tarballs
> of our workspace to customers, thinking it would save something, but it
> appears the savings are limited, and now I have a customer complaint about
> this, since I have one recipe that uses a private repository.
>
> What is the logic of not using the downloads directory for the source of
> the unpack?
>
> I recently fixed a recipe by writing an unpack that does not use
> networking and does a clone of the downloads directory to get around this.
> Why is this not done by openembedded-core?
>
> Thanks.
>
>
[-- Attachment #2: Type: text/html, Size: 1469 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] git do_unpack
2020-06-28 19:30 ` [OE-core] " Alexander Kanavin
@ 2020-06-30 4:45 ` John Klug
2020-06-30 8:23 ` Quentin Schulz
2020-06-30 8:24 ` Alexander Kanavin
0 siblings, 2 replies; 7+ messages in thread
From: John Klug @ 2020-06-30 4:45 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: OE-core
a:
(fetch was already run successfully while connected with a VPN).
VPN then shut down.
$ bitbake -c unpack dhq-client
...
NOTE: Executing RunQueue Tasks
ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1"; export SSH_AGENT_PID="2444"; export SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git failed with exit code 128, output:
ssh: connect to host gitlab.multitech.net port 22: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
ERROR: Task (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
Line 3 of the do_unpack.log:
DEBUG: Fetcher accessed the network with the command git -c core.fsyncobjectfiles=0 ls-remote ssh://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git
Testing the download of the git (download is verifiable):
klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$ git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (485/485), done.
klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$
b. Too busy writing my own git commands in my own do_unpack to fix this. Maybe someday.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] git do_unpack
2020-06-30 4:45 ` John Klug
@ 2020-06-30 8:23 ` Quentin Schulz
2020-06-30 8:24 ` Alexander Kanavin
1 sibling, 0 replies; 7+ messages in thread
From: Quentin Schulz @ 2020-06-30 8:23 UTC (permalink / raw)
To: John Klug; +Cc: Alexander Kanavin, OE-core
Hi John,
On Tue, Jun 30, 2020 at 04:45:32AM +0000, John Klug wrote:
>
> a:
>
> (fetch was already run successfully while connected with a VPN).
>
> VPN then shut down.
>
> $ bitbake -c unpack dhq-client
>
> ...
> NOTE: Executing RunQueue Tasks
> ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1"; export SSH_AGENT_PID="2444"; export SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git failed with exit code 128, output:
> ssh: connect to host gitlab.multitech.net port 22: Connection timed out
> fatal: Could not read from remote repository.
>
> Please make sure you have the correct access rights
> and the repository exists.
>
> ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
> ERROR: Logfile of failure stored in: /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
> ERROR: Task (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack) failed with exit code '1'
> NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.
>
> Summary: 1 task failed:
> /home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack
> Summary: There was 1 WARNING message shown.
> Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
>
>
> Line 3 of the do_unpack.log:
>
> DEBUG: Fetcher accessed the network with the command git -c core.fsyncobjectfiles=0 ls-remote ssh://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git
>
Any chance you're using a tag in your SRC_URI?
https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#git-fetcher
or AUTOREV? Which I guess needs network access to know if there's
something to pull or not.
Quentin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] git do_unpack
2020-06-30 4:45 ` John Klug
2020-06-30 8:23 ` Quentin Schulz
@ 2020-06-30 8:24 ` Alexander Kanavin
2020-06-30 17:24 ` John Klug
1 sibling, 1 reply; 7+ messages in thread
From: Alexander Kanavin @ 2020-06-30 8:24 UTC (permalink / raw)
To: John Klug; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 3483 bytes --]
This happens if the recipe does not specify the exact revision hash to use
in SRCREV. If it says "use latest revision" or "use this tag", then
bitbake will do ls-remote on the original repo to resolve those to
up-to-date revisions, which is not unreasonable.
What does the recipe look like?
Alex
On Tue, 30 Jun 2020 at 06:45, John Klug <John.Klug@multitech.com> wrote:
>
> a:
>
> (fetch was already run successfully while connected with a VPN).
>
> VPN then shut down.
>
> $ bitbake -c unpack dhq-client
>
> ...
> NOTE: Executing RunQueue Tasks
> ERROR: dhq-client-0.32-r11 do_unpack: Fetcher failure: Fetch command
> export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-CbyhEfsmW1";
> export SSH_AGENT_PID="2444"; export
> SSH_AUTH_SOCK="/tmp/ssh-gHgQLFqq7E5R/agent.2443"; export
> PATH="/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/python-native:/home/klugja/mlinux-up-14/layers/openembedded-core/scripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin/arm-mlinux-linux-gnueabi:/home/klugja/mlinux-up-14/build/tmp/sysroots/mt100eocg/usr/bin/crossscripts:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/usr/bin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/sbin:/home/klugja/mlinux-up-14/build/tmp/sysroots/x86_64-linux/bin:/home/klugja/mlinux-up-14/scripts:/home/klugja/mlinux-up-14/bitbake/bin:/home/klugja/bin:/home/klugja/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin";
> export HOME="/home/klugja"; git -c core.fsyncobjectfiles=0 ls-remote ssh://
> git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git failed with exit
> code 128, output:
> ssh: connect to host gitlab.multitech.net port 22: Connection timed out
> fatal: Could not read from remote repository.
>
> Please make sure you have the correct access rights
> and the repository exists.
>
> ERROR: dhq-client-0.32-r11 do_unpack: Function failed: base_do_unpack
> ERROR: Logfile of failure stored in:
> /home/klugja/mlinux-up-14/build/tmp/work/mt100eocg-mlinux-linux-gnueabi/dhq-client/0.32-r11/temp/log.do_unpack.10175
> ERROR: Task
> (/home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:do_unpack)
> failed with exit code '1'
> NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be
> rerun and 1 failed.
>
> Summary: 1 task failed:
>
> /home/klugja/mlinux-up-14/build/../layers/meta-mlinux/recipes-core/dhq-client/dhq-client_0.32.bb:
> do_unpack
> Summary: There was 1 WARNING message shown.
> Summary: There were 2 ERROR messages shown, returning a non-zero exit
> code.
>
>
> Line 3 of the do_unpack.log:
>
> DEBUG: Fetcher accessed the network with the command git -c
> core.fsyncobjectfiles=0 ls-remote ssh://
> git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git
>
> Testing the download of the git (download is verifiable):
> klugja@peru:~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$
> git fsck
> Checking object directories: 100% (256/256), done.
> Checking objects: 100% (485/485), done.
> klugja@peru
> :~/mlinux-up-14/build/downloads/git2/gitlab.multitech.net.devicehq.mAPI-dhq-client.git$
>
>
>
> b. Too busy writing my own git commands in my own do_unpack to fix this.
> Maybe someday.
>
[-- Attachment #2: Type: text/html, Size: 4265 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] git do_unpack
2020-06-30 8:24 ` Alexander Kanavin
@ 2020-06-30 17:24 ` John Klug
2020-06-30 17:31 ` Alexander Kanavin
0 siblings, 1 reply; 7+ messages in thread
From: John Klug @ 2020-06-30 17:24 UTC (permalink / raw)
To: Alexander Kanavin, OE-core
We typically use tags to match PV and name our recipes NAME_TAG.bb
Recipe name:
dhq-client_0.39.bb
from bitbake -e dhq-client output:
# pre-expansion value:
# "${PN}-${PV}"
P="dhq-client-0.39"
In recipe:
SRCREV = "${PV}"
SRC_URI = "git://git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git;protocol=ssh;nobranch=1 \ file://config.json.sample \ file://dhqclient.init \ file://dhqclient \ file://dhq-rootCA.pem \ "
Sanity check:
jklug@trustytahrdm1:~$ git clone git@gitlab.multitech.net:devicehq/mAPI-dhq-client
Cloning into 'mAPI-dhq-client'...
remote: Counting objects: 485, done.
remote: Compressing objects: 100% (199/199), done.
remote: Total 485 (delta 285), reused 450 (delta 265)
Receiving objects: 100% (485/485), 84.73 KiB | 14.12 MiB/s, done.
Resolving deltas: 100% (285/285), done.
jklug@trustytahrdm1:~$ cd mAPI-dhq-client/
jklug@trustytahrdm1:~/mAPI-dhq-client$ git log -1 --format=%ai 0.39 | head
2020-05-05 09:47:03 -0500
jklug@trustytahrdm1:~/mAPI-dhq-client$ git checkout 0.39
Note: checking out '0.39'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at d21aab3 Merge branch 'vp/fix_logs' into 'master'
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] git do_unpack
2020-06-30 17:24 ` John Klug
@ 2020-06-30 17:31 ` Alexander Kanavin
0 siblings, 0 replies; 7+ messages in thread
From: Alexander Kanavin @ 2020-06-30 17:31 UTC (permalink / raw)
To: John Klug; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 1897 bytes --]
SRCREV should be the actual commit hash that matches the tag, then
do_unpack will not do git ls-remote. Tags can, and do move around.
Alex
On Tue, 30 Jun 2020 at 19:24, John Klug <John.Klug@multitech.com> wrote:
> We typically use tags to match PV and name our recipes NAME_TAG.bb
>
> Recipe name:
>
> dhq-client_0.39.bb
>
> from bitbake -e dhq-client output:
>
> # pre-expansion value:
> # "${PN}-${PV}"
> P="dhq-client-0.39"
>
> In recipe:
>
> SRCREV = "${PV}"
> SRC_URI = "git://
> git@gitlab.multitech.net/devicehq/mAPI-dhq-client.git;protocol=ssh;nobranch=1
> \ file://config.json.sample \ file://dhqclient.init
> \ file://dhqclient \ file://dhq-rootCA.pem \ "
>
> Sanity check:
>
> jklug@trustytahrdm1:~$ git clone git@gitlab.multitech.net:
> devicehq/mAPI-dhq-client
> Cloning into 'mAPI-dhq-client'...
> remote: Counting objects: 485, done.
> remote: Compressing objects: 100% (199/199), done.
> remote: Total 485 (delta 285), reused 450 (delta 265)
> Receiving objects: 100% (485/485), 84.73 KiB | 14.12 MiB/s, done.
> Resolving deltas: 100% (285/285), done.
> jklug@trustytahrdm1:~$ cd mAPI-dhq-client/
> jklug@trustytahrdm1:~/mAPI-dhq-client$ git log -1 --format=%ai 0.39 | head
> 2020-05-05 09:47:03 -0500
> jklug@trustytahrdm1:~/mAPI-dhq-client$ git checkout 0.39
> Note: checking out '0.39'.
>
> You are in 'detached HEAD' state. You can look around, make experimental
> changes and commit them, and you can discard any commits you make in this
> state without impacting any branches by performing another checkout.
>
> If you want to create a new branch to retain commits you create, you may
> do so (now or later) by using -b with the checkout command again. Example:
>
> git checkout -b <new-branch-name>
>
> HEAD is now at d21aab3 Merge branch 'vp/fix_logs' into 'master'
>
[-- Attachment #2: Type: text/html, Size: 2614 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-06-30 17:31 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-28 17:27 git do_unpack John Klug
2020-06-28 19:30 ` [OE-core] " Alexander Kanavin
2020-06-30 4:45 ` John Klug
2020-06-30 8:23 ` Quentin Schulz
2020-06-30 8:24 ` Alexander Kanavin
2020-06-30 17:24 ` John Klug
2020-06-30 17:31 ` Alexander Kanavin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.