[PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

[PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

[Aaron Chan]

This patch is to fix the inconsistency in loading custom module
yoctoabb & yocto_console_view during Buildbot Master startup.

Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
---
 __init__.py                    | 0
 yocto_console_view/__init__.py | 0
 2 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 __init__.py
 create mode 100644 yocto_console_view/__init__.py

diff --git a/__init__.py b/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/yocto_console_view/__init__.py b/yocto_console_view/__init__.py
new file mode 100644
index 0000000..e69de29
-- 
2.7.4

Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

[Richard Purdie]

On Tue, 2018-07-10 at 11:24 +0800, Aaron Chan wrote:
> This patch is to fix the inconsistency in loading custom module
> yoctoabb & yocto_console_view during Buildbot Master startup.
> 
> Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
> ---
>  __init__.py                    | 0
>  yocto_console_view/__init__.py | 0
>  2 files changed, 0 insertions(+), 0 deletions(-)
>  create mode 100644 __init__.py
>  create mode 100644 yocto_console_view/__init__.py

I think this means you're using python2 and we really should be using
python3 as I don't want to support both...

Cheers,

Richard

Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

[Chan, Aaron Chun Yew]

Hi Richard,

[Richard] I think this means you're using python2 and we really should be using
                python3 as I don't want to support both...

[Reply] This error/bug was found during buildbot startup meaning this is out of my control. 
            Maybe you have a fix for this, otherwise I do suggest to add in __init__.py in places where we need to source custom modules.

$ buildbot start ~/yocto-controller
$ cat -n 100 ~/yocto-controller/twistd.log

5800	2018-07-10 18:42:13+0800 [-] Main loop terminated.
  5801	2018-07-10 18:42:13+0800 [-] Server Shut Down.
  5802	2018-07-10 18:42:27+0800 [-] Loading buildbot.tac...
  5803	2018-07-10 18:42:27+0800 [-] Loaded.
  5804	2018-07-10 18:42:27+0800 [-] twistd 18.4.0 (/usr/bin/python 2.7.13) starting up.
  5805	2018-07-10 18:42:27+0800 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
  5806	2018-07-10 18:42:27+0800 [-] Starting BuildMaster -- buildbot.version: 1.2.0
  5807	2018-07-10 18:42:27+0800 [-] Loading configuration from '/home/ab/yocto-controller/master.cfg'
  5808	2018-07-10 18:42:27+0800 [-] error while parsing config file:
  5809		Traceback (most recent call last):
  5810		  File "/home/ab/.local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
  5811		    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  5812		  File "/home/ab/.local/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
  5813		    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  5814		  File "/home/ab/.local/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
  5815		    return func(*args,**kw)
  5816		  File "/home/ab/.local/lib/python2.7/site-packages/buildbot/config.py", line 182, in loadConfig
  5817		    self.basedir, self.configFileName)
  5818		--- <exception caught here> ---
  5819		  File "/home/ab/.local/lib/python2.7/site-packages/buildbot/config.py", line 140, in loadConfigDict
  5820		    execfile(filename, localDict)
  5821		  File "/home/ab/.local/lib/python2.7/site-packages/twisted/python/compat.py", line 246, in execfile
  5822		    exec(code, globals, locals)
  5823		  File "/home/ab/yocto-controller/master.cfg", line 11, in <module>
  5824		    from yoctoabb import builders, config, schedulers, workers, services, www
  5825		exceptions.ImportError: No module named yoctoabb
  5826		
  5827	2018-07-10 18:42:27+0800 [-] Configuration Errors:
  5828	2018-07-10 18:42:27+0800 [-]   error while parsing config file: No module named yoctoabb (traceback in logfile)
  5829	2018-07-10 18:42:27+0800 [-] Halting master.
  5830	2018-07-10 18:42:27+0800 [-] BuildMaster startup failed
  5831	2018-07-10 18:42:27+0800 [-] BuildMaster is stopped
  5832	2018-07-10 18:42:27+0800 [-] Main loop terminated.
  5833	2018-07-10 18:42:27+0800 [-] Server Shut Down.

Cheers,
Aaron
________________________________________
From: richard.purdie@linuxfoundation.org [richard.purdie@linuxfoundation.org]
Sent: Tuesday, July 10, 2018 4:40 PM
To: Chan, Aaron Chun Yew; yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
Subject: Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

On Tue, 2018-07-10 at 11:24 +0800, Aaron Chan wrote:
> This patch is to fix the inconsistency in loading custom module
> yoctoabb & yocto_console_view during Buildbot Master startup.
>
> Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
> ---
>  __init__.py                    | 0
>  yocto_console_view/__init__.py | 0
>  2 files changed, 0 insertions(+), 0 deletions(-)
>  create mode 100644 __init__.py
>  create mode 100644 yocto_console_view/__init__.py

I think this means you're using python2 and we really should be using
python3 as I don't want to support both...

Cheers,

Richard

Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

[Richard Purdie]

On Tue, 2018-07-10 at 10:47 +0000, Chan, Aaron Chun Yew wrote:
> [Richard] I think this means you're using python2 and we really
> should be using
>                 python3 as I don't want to support both...
> 
> [Reply] This error/bug was found during buildbot startup meaning this
> is out of my control.

No, it is not out of your control.

>             Maybe you have a fix for this, otherwise I do suggest to
> add in __init__.py in places where we need to source custom modules.
> 
> $ buildbot start ~/yocto-controller
> $ cat -n 100 ~/yocto-controller/twistd.log

Try using virtualenv:

$ python --version
Python 2.7.15rc1

$ virtualenv -p python3 test
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /media/build1/test/bin/python3
Also creating executable in /media/build1/test/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

$ . ./test/bin/activate

(test) richard@dax:/media/build1/$ python --version
Python 3.6.5


If you start buildbot within the virtualenv, you should see it using
python3.

Cheers,

Richard

Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

[Chan, Aaron Chun Yew]

Hi Richard,

It appears that your using virtualenv (switch to python3) to startup your buildbot processes and not using the default python from host distribution.
I am aware that you can switch interpreter to python2/3 in virtualenv. Shouldn’t we handle these for both in events users uses
the default host distribution as default python2 and still make it worked? What is the best practices here and should all the buildbot-workers start in
virtualenv defaults to python3 as well ? I am little lost right here, please advise.

Cheers,
Aaron

Try using virtualenv:

$ python --version
Python 2.7.15rc1

$ virtualenv -p python3 test
Already using interpreter /usr/bin/python3 Using base prefix '/usr'
New python executable in /media/build1/test/bin/python3 Also creating executable in /media/build1/test/bin/python Installing setuptools, pkg_resources, pip, wheel...done.

$ . ./test/bin/activate

(test) richard@dax:/media/build1/$ python --version Python 3.6.5


If you start buildbot within the virtualenv, you should see it using python3.

Cheers,

Richard

-----Original Message-----
From: richard.purdie@linuxfoundation.org [mailto:richard.purdie@linuxfoundation.org] 
Sent: Wednesday, July 11, 2018 6:17 AM
To: Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com>; yocto@yoctoproject.org; Burton, Ross <ross.burton@intel.com>; Eggleton, Paul <paul.eggleton@intel.com>
Subject: Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

On Tue, 2018-07-10 at 10:47 +0000, Chan, Aaron Chun Yew wrote:
> [Richard] I think this means you're using python2 and we really should
> be using
>                 python3 as I don't want to support both...
> 
> [Reply] This error/bug was found during buildbot startup meaning this 
> is out of my control.

No, it is not out of your control.

>             Maybe you have a fix for this, otherwise I do suggest to 
> add in __init__.py in places where we need to source custom modules.
> 
> $ buildbot start ~/yocto-controller
> $ cat -n 100 ~/yocto-controller/twistd.log

Try using virtualenv:

$ python --version
Python 2.7.15rc1

$ virtualenv -p python3 test
Already using interpreter /usr/bin/python3 Using base prefix '/usr'
New python executable in /media/build1/test/bin/python3 Also creating executable in /media/build1/test/bin/python Installing setuptools, pkg_resources, pip, wheel...done.

$ . ./test/bin/activate

(test) richard@dax:/media/build1/$ python --version Python 3.6.5


If you start buildbot within the virtualenv, you should see it using python3.

Cheers,

Richard

Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

[Richard Purdie]

On Wed, 2018-07-11 at 02:23 +0000, Chan, Aaron Chun Yew wrote:
> It appears that your using virtualenv (switch to python3) to startup
> your buildbot processes and not using the default python from host
> distribution.
> I am aware that you can switch interpreter to python2/3 in
> virtualenv. Shouldn’t we handle these for both in events users uses
> the default host distribution as default python2 and still make it
> worked? What is the best practices here and should all the buildbot-
> workers start in
> virtualenv defaults to python3 as well ? I am little lost right here,
> please advise.

There are multiple ways to achieve this, you can make /usr/bin/python
point at python3 too, or simply not install python2 on the controller.
Using virtualenv was just a simple way to illustrate how you can use
it.

My worry about trying to support both python versions is that as you've
found out, there are problems that can occur in one that don't occur in
the other, which then means twice as much testing. Since python2 is an
official "dead end" and as a project we're trying hard to use python3
everywhere, it doesn't seem to make sense to try and support python2 on
new code such as the autobuilder?

Cheers,

Richard

Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

[Chan, Aaron Chun Yew]

Hi Richard,

Thanks for the well crafted explanation, and after talking to Paul I got a clearer picture on which direction we should be heading and we should align
and work towards python3. Maybe I was not aware, my bad.

Cheers,
Aaron

-----Original Message-----
From: richard.purdie@linuxfoundation.org [mailto:richard.purdie@linuxfoundation.org] 
Sent: Wednesday, July 11, 2018 3:12 PM
To: Chan, Aaron Chun Yew <aaron.chun.yew.chan@intel.com>; yocto@yoctoproject.org; Burton, Ross <ross.burton@intel.com>; Eggleton, Paul <paul.eggleton@intel.com>
Subject: Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view

On Wed, 2018-07-11 at 02:23 +0000, Chan, Aaron Chun Yew wrote:
> It appears that your using virtualenv (switch to python3) to startup 
> your buildbot processes and not using the default python from host 
> distribution.
> I am aware that you can switch interpreter to python2/3 in virtualenv. 
> Shouldn’t we handle these for both in events users uses the default 
> host distribution as default python2 and still make it worked? What is 
> the best practices here and should all the buildbot- workers start in 
> virtualenv defaults to python3 as well ? I am little lost right here, 
> please advise.

There are multiple ways to achieve this, you can make /usr/bin/python point at python3 too, or simply not install python2 on the controller.
Using virtualenv was just a simple way to illustrate how you can use it.

My worry about trying to support both python versions is that as you've found out, there are problems that can occur in one that don't occur in the other, which then means twice as much testing. Since python2 is an official "dead end" and as a project we're trying hard to use python3 everywhere, it doesn't seem to make sense to try and support python2 on new code such as the autobuilder?

Cheers,

Richard