Toaster Not Respecting Environment Variables

Toaster Not Respecting Environment Variables

[Gillespie, Bryan]

[-- Attachment #1: Type: text/plain, Size: 2223 bytes --]

All,


I am trying to use Toaster to configure and build project, however, I am working behind a corporate firewall that requires a proxy. When I try to build core-image-minimal with Toaster, it fails immediately with:


2017-01-17 11:22:44,942 ERROR runbuilds: Error launching build command: git fetch --all && git reset --hard "origin/krogoth"
fatal: Unable to look up git.yoctoproject.org (port 9418) (Temporary failure in name resolution)
error: Could not fetch origin

Traceback (most recent call last):
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py", line 67, in schedule
    "%d:%d" % (br.pk, bec.be.pk))
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 255, in triggerBuild
    layers = self.setLayers(bitbake, layers, targets)
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 161, in setLayers
    self._shellcmd('git fetch --all && git reset --hard "%s"' % ref, localdirname)
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 70, in _shellcmd
    raise ShellCmdException(err)
ShellCmdException: command: git fetch --all && git reset --hard "origin/krogoth"
fatal: Unable to look up git.yoctoproject.org (port 9418) (Temporary failure in name resolution)
error: Could not fetch origin



Two questions:

1. Why is it using git protocol (port 9418)? My origin remote for the project is setup using https:


(venv) user@mantis:~/poky/build$ git remote -v
origin https://git.yoctoproject.org/git/poky (fetch)
origin https://git.yoctoproject.org/git/poky (push)


Is it hardcoded? I prefer https because it is much faster in our network setup.


2. That command definitely works when the proxy environment variables are set. I added the following snippet to  bitbake/lib/toaster/bldcontrol/localhostbecontroller.py:


logger.debug("ENV::",os.environ)


The output of that log indicates to me that Toaster is purging all environment variables, hence why the git fetch fails.


Should I file a bug report or am I just missing something?


Thanks,

Bryan


=========================
Lab: x3214, Office: x5956

[-- Attachment #2: Type: text/html, Size: 3625 bytes --]

Re: Toaster Not Respecting Environment Variables

[Reyna, David]

[-- Attachment #1: Type: text/plain, Size: 3911 bytes --]

Hi Bryan,

Thank you for your question.

  1. Git is not hard coded to use the git port in Toaster. It just uses standard "git clone" commands (in toaster/bldcontrol/localhostbecontroller.py" specifically). But see the answer to #2.

  2. The local environment is indeed purged/ignored by Toaster. The reason was that for the remote build aspect of Toaster we wanted the person who was the "build master" to control the environment over remote users, so that the builds were safe. I understand your point that this is a problem for local usage of Toaster, where you are your own build master.

There is actually a defect around this topic which may be of interest:
  Bug 10281<https://bugzilla.yoctoproject.org/show_bug.cgi?id=10281> - Configuration variables: I can set a value for standard shell environment variable http_proxy

What is interesting about this defect is that this shows you how you can actually get around the limitation, and specifically for "http_proxy", by setting it as a Toaster/bitbake environment variable. Obviously before we "fix" this defect (to close this loophole) we should have a general solution for supporting local builders like yourself.

Also, by explicitly setting this variable in Toaster builds, remote builds across the web will still get the proper value set, as per the goal that Toaster it is not dependent on (or vulnerable to) your local environment.

- David

From: toaster-bounces@yoctoproject.org [mailto:toaster-bounces@yoctoproject.org] On Behalf Of Gillespie, Bryan
Sent: Tuesday, January 17, 2017 8:40 AM
To: toaster@yoctoproject.org
Subject: [Toaster] Toaster Not Respecting Environment Variables


All,



I am trying to use Toaster to configure and build project, however, I am working behind a corporate firewall that requires a proxy. When I try to build core-image-minimal with Toaster, it fails immediately with:


2017-01-17 11:22:44,942 ERROR runbuilds: Error launching build command: git fetch --all && git reset --hard "origin/krogoth"
fatal: Unable to look up git.yoctoproject.org (port 9418) (Temporary failure in name resolution)
error: Could not fetch origin

Traceback (most recent call last):
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py", line 67, in schedule
    "%d:%d" % (br.pk, bec.be.pk))
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 255, in triggerBuild
    layers = self.setLayers(bitbake, layers, targets)
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 161, in setLayers
    self._shellcmd('git fetch --all && git reset --hard "%s"' % ref, localdirname)
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 70, in _shellcmd
    raise ShellCmdException(err)
ShellCmdException: command: git fetch --all && git reset --hard "origin/krogoth"
fatal: Unable to look up git.yoctoproject.org (port 9418) (Temporary failure in name resolution)
error: Could not fetch origin




Two questions:

1. Why is it using git protocol (port 9418)? My origin remote for the project is setup using https:


(venv) user@mantis:~/poky/build$ git remote -v
origin https://git.yoctoproject.org/git/poky (fetch)
origin https://git.yoctoproject.org/git/poky (push)


Is it hardcoded? I prefer https because it is much faster in our network setup.



2. That command definitely works when the proxy environment variables are set. I added the following snippet to  bitbake/lib/toaster/bldcontrol/localhostbecontroller.py:


logger.debug("ENV::",os.environ)



The output of that log indicates to me that Toaster is purging all environment variables, hence why the git fetch fails.



Should I file a bug report or am I just missing something?



Thanks,

Bryan


=========================
Lab: x3214, Office: x5956

[-- Attachment #2: Type: text/html, Size: 14545 bytes --]

Re: Toaster Not Respecting Environment Variables

[Gillespie, Bryan]

[-- Attachment #1: Type: text/plain, Size: 4612 bytes --]

David,


Thanks for the reply. I was able to get local toaster builds working using the "defect"/workaround + .gitconfig find/replace (to switch all git:// requests to be https:// requests... not sure if there is a more standard way of telling yocto to use https git urls).


Eventually we probably will have a remote build VM, so I think this was only a temporary setback.


Thanks again!

Bryan


=========================
Lab: x3214, Office: x5956
________________________________
From: Reyna, David <david.reyna@windriver.com>
Sent: Friday, January 20, 2017 4:40:44 PM
To: Gillespie, Bryan; toaster@yoctoproject.org
Subject: RE: Toaster Not Respecting Environment Variables

Hi Bryan,

Thank you for your question.

  1. Git is not hard coded to use the git port in Toaster. It just uses standard "git clone" commands (in toaster/bldcontrol/localhostbecontroller.py" specifically). But see the answer to #2.

  2. The local environment is indeed purged/ignored by Toaster. The reason was that for the remote build aspect of Toaster we wanted the person who was the "build master" to control the environment over remote users, so that the builds were safe. I understand your point that this is a problem for local usage of Toaster, where you are your own build master.

There is actually a defect around this topic which may be of interest:
  Bug 10281<https://bugzilla.yoctoproject.org/show_bug.cgi?id=10281> - Configuration variables: I can set a value for standard shell environment variable http_proxy

What is interesting about this defect is that this shows you how you can actually get around the limitation, and specifically for "http_proxy", by setting it as a Toaster/bitbake environment variable. Obviously before we "fix" this defect (to close this loophole) we should have a general solution for supporting local builders like yourself.

Also, by explicitly setting this variable in Toaster builds, remote builds across the web will still get the proper value set, as per the goal that Toaster it is not dependent on (or vulnerable to) your local environment.

- David

From: toaster-bounces@yoctoproject.org [mailto:toaster-bounces@yoctoproject.org] On Behalf Of Gillespie, Bryan
Sent: Tuesday, January 17, 2017 8:40 AM
To: toaster@yoctoproject.org
Subject: [Toaster] Toaster Not Respecting Environment Variables


All,



I am trying to use Toaster to configure and build project, however, I am working behind a corporate firewall that requires a proxy. When I try to build core-image-minimal with Toaster, it fails immediately with:


2017-01-17 11:22:44,942 ERROR runbuilds: Error launching build command: git fetch --all && git reset --hard "origin/krogoth"
fatal: Unable to look up git.yoctoproject.org (port 9418) (Temporary failure in name resolution)
error: Could not fetch origin

Traceback (most recent call last):
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py", line 67, in schedule
    "%d:%d" % (br.pk, bec.be.pk))
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 255, in triggerBuild
    layers = self.setLayers(bitbake, layers, targets)
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 161, in setLayers
    self._shellcmd('git fetch --all && git reset --hard "%s"' % ref, localdirname)
  File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 70, in _shellcmd
    raise ShellCmdException(err)
ShellCmdException: command: git fetch --all && git reset --hard "origin/krogoth"
fatal: Unable to look up git.yoctoproject.org (port 9418) (Temporary failure in name resolution)
error: Could not fetch origin




Two questions:

1. Why is it using git protocol (port 9418)? My origin remote for the project is setup using https:


(venv) user@mantis:~/poky/build$ git remote -v
origin https://git.yoctoproject.org/git/poky (fetch)
origin https://git.yoctoproject.org/git/poky (push)


Is it hardcoded? I prefer https because it is much faster in our network setup.



2. That command definitely works when the proxy environment variables are set. I added the following snippet to  bitbake/lib/toaster/bldcontrol/localhostbecontroller.py:


logger.debug("ENV::",os.environ)



The output of that log indicates to me that Toaster is purging all environment variables, hence why the git fetch fails.



Should I file a bug report or am I just missing something?



Thanks,

Bryan


=========================
Lab: x3214, Office: x5956

[-- Attachment #2: Type: text/html, Size: 15970 bytes --]

Re: Toaster Not Respecting Environment Variables

[Brian Avery]

[-- Attachment #1: Type: text/plain, Size: 5535 bytes --]

Sorry for the late reply.  Here's the wiki page on how work with Yocto
behind a proxy.  The oe-git-proxy should work with toaster.
https://wiki.yoctoproject.org/wiki/Working_Behind_a_Network_Proxy
-brian
an intel employee

On Fri, Jan 20, 2017 at 4:19 PM, Gillespie, Bryan <
Bryan.Gillespie@hughes.com> wrote:

> David,
>
>
> Thanks for the reply. I was able to get local toaster builds working using
> the "defect"/workaround + .gitconfig find/replace (to switch all git://
> requests to be https:// requests... not sure if there is a more standard
> way of telling yocto to use https git urls).
>
>
> Eventually we probably will have a remote build VM, so I think this was
> only a temporary setback.
>
>
> Thanks again!
>
> Bryan
>
>
> =========================
> Lab: x3214, Office: x5956
> ------------------------------
> *From:* Reyna, David <david.reyna@windriver.com>
> *Sent:* Friday, January 20, 2017 4:40:44 PM
> *To:* Gillespie, Bryan; toaster@yoctoproject.org
> *Subject:* RE: Toaster Not Respecting Environment Variables
>
>
> Hi Bryan,
>
>
>
> Thank you for your question.
>
>
>
>   1. Git is not hard coded to use the git port in Toaster. It just uses
> standard “git clone” commands (in toaster/bldcontrol/localhostbecontroller.py”
> specifically). But see the answer to #2.
>
>
>
>   2. The local environment is indeed purged/ignored by Toaster. The reason
> was that for the remote build aspect of Toaster we wanted the person who
> was the “build master” to control the environment over remote users, so
> that the builds were safe. I understand your point that this is a problem
> for local usage of Toaster, where you are your own build master.
>
>
>
> There is actually a defect around this topic which may be of interest:
>
>   *Bug 10281* <https://bugzilla.yoctoproject.org/show_bug.cgi?id=10281> -
> Configuration variables: I can set a value for standard shell environment
> variable http_proxy
>
>
>
> What is interesting about this defect is that this shows you how you can
> actually get around the limitation, and specifically for “http_proxy”, by
> setting it as a Toaster/bitbake environment variable. Obviously before we
> “fix” this defect (to close this loophole) we should have a general
> solution for supporting local builders like yourself.
>
>
>
> Also, by explicitly setting this variable in Toaster builds, remote builds
> across the web will still get the proper value set, as per the goal that
> Toaster it is not dependent on (or vulnerable to) your local environment.
>
>
>
> - David
>
>
>
> *From:* toaster-bounces@yoctoproject.org [mailto:toaster-bounces@
> yoctoproject.org] *On Behalf Of *Gillespie, Bryan
> *Sent:* Tuesday, January 17, 2017 8:40 AM
> *To:* toaster@yoctoproject.org
> *Subject:* [Toaster] Toaster Not Respecting Environment Variables
>
>
>
> All,
>
>
>
> I am trying to use Toaster to configure and build project, however, I am
> working behind a corporate firewall that requires a proxy. When I try to
> build core-image-minimal with Toaster, it fails immediately with:
>
>
>
> 2017-01-17 11:22:44,942 ERROR runbuilds: Error launching build command:
> git fetch --all && git reset --hard "origin/krogoth"
>
> fatal: Unable to look up git.yoctoproject.org (port 9418) (Temporary
> failure in name resolution)
>
> error: Could not fetch origin
>
>
>
> Traceback (most recent call last):
>
>   File "/home/user/poky/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py",
> line 67, in schedule
>
>     "%d:%d" % (br.pk, bec.be.pk))
>
>   File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py",
> line 255, in triggerBuild
>
>     layers = self.setLayers(bitbake, layers, targets)
>
>   File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py",
> line 161, in setLayers
>
>     self._shellcmd('git fetch --all && git reset --hard "%s"' % ref,
> localdirname)
>
>   File "/home/user/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py",
> line 70, in _shellcmd
>
>     raise ShellCmdException(err)
>
> ShellCmdException: command: git fetch --all && git reset --hard
> "origin/krogoth"
>
> fatal: Unable to look up git.yoctoproject.org (port 9418) (Temporary
> failure in name resolution)
>
> error: Could not fetch origin
>
>
>
>
>
> Two questions:
>
> 1. Why is it using git protocol (port 9418)? My origin remote for the
> project is setup using https:
>
>
>
> (venv) user@mantis:~/poky/build$ git remote -v
>
> origin https://git.yoctoproject.org/git/poky (fetch)
>
> origin https://git.yoctoproject.org/git/poky (push)
>
>
>
> Is it hardcoded? I prefer https because it is much faster in our network
> setup.
>
>
>
> 2. That command definitely works when the proxy environment variables are
> set. I added the following snippet to  bitbake/lib/toaster/bldcontrol/
> localhostbecontroller.py:
>
>
>
> logger.debug("ENV::",os.environ)
>
>
>
> The output of that log indicates to me that Toaster is purging all
> environment variables, hence why the git fetch fails.
>
>
>
> Should I file a bug report or am I just missing something?
>
>
>
> Thanks,
>
> Bryan
>
>
>
> =========================
> Lab: x3214, Office: x5956
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>
>

[-- Attachment #2: Type: text/html, Size: 16166 bytes --]