[PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

[PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

[Aaron Chan]

This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
The socket module allows the retrieval on FQDN and constructs the entire
URL by default, this default settings can be overwritten in c['buildbotURL']
based on local administrator preferences.

Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
---
 master.cfg | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/master.cfg b/master.cfg
index fca80d2..49ddeb4 100644
--- a/master.cfg
+++ b/master.cfg
@@ -4,6 +4,7 @@
 import os
 import imp
 import pkg_resources
+import socket
 
 from buildbot.plugins import *
 from buildbot.plugins import db
@@ -55,6 +56,7 @@ imp.reload(services)
 imp.reload(www)
 
 c = BuildmasterConfig = {}
+url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port) + '/')
 
 # Disable usage reporting
 c['buildbotNetUsageData'] = None
@@ -76,6 +78,7 @@ c['www'] = www.www
 c['workers'] = workers.workers
 
 c['title'] = "Yocto Autobuilder"
-c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
+c['titleURL'] = url
 # visible location for internal web server
-c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
+# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
+c['buildbotURL'] = url
-- 
2.7.4

Re: [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

[Martin Hundebøll]

Hi Aron,

On 2018-07-10 05:18, Aaron Chan wrote:
> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
> The socket module allows the retrieval on FQDN and constructs the entire
> URL by default, this default settings can be overwritten in c['buildbotURL']
> based on local administrator preferences.
> 
> Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
> ---
>   master.cfg | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/master.cfg b/master.cfg
> index fca80d2..49ddeb4 100644
> --- a/master.cfg
> +++ b/master.cfg
> @@ -4,6 +4,7 @@
>   import os
>   import imp
>   import pkg_resources
> +import socket
>   
>   from buildbot.plugins import *
>   from buildbot.plugins import db
> @@ -55,6 +56,7 @@ imp.reload(services)
>   imp.reload(www)
>   
>   c = BuildmasterConfig = {}
> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port) + '/')

Why use `os.path.join()` here? It isn't supposed to be used to construct 
url's, and is overkill for this case, and you'd end up with "http:///...".

// Martin

>   
>   # Disable usage reporting
>   c['buildbotNetUsageData'] = None
> @@ -76,6 +78,7 @@ c['www'] = www.www
>   c['workers'] = workers.workers
>   
>   c['title'] = "Yocto Autobuilder"
> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
> +c['titleURL'] = url
>   # visible location for internal web server
> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
> +c['buildbotURL'] = url
> 

-- 
Kind regards,
Martin Hundebøll
Embedded Linux Consultant

+45 61 65 54 61
martin@geanix.com

Geanix IVS
DK39600706

Re: [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

[Chan, Aaron Chun Yew]

My name is Aaron and not Aron for start

Martin,

Please try this

#!/usr/bin/env python2

import os

a = os.path.join('http://', 'alibaba.com')
b = '/'.join(['http://', 'alibaba.com'])
c = '/'.join(['http:/', 'alibaba.com', ''])

print(a)
print(b)
print(c)

and repeat the same for python3, I got the following results:

http://alibaba.com
http:///alibaba.com
http://alibaba.com/

Cheers,
Aaron
________________________________________
From: Martin Hundebøll [martin@geanix.com]
Sent: Tuesday, July 10, 2018 2:10 PM
To: Chan, Aaron Chun Yew; richard.purdie@linuxfoundation.org; yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
Subject: Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

Hi Aron,

On 2018-07-10 05:18, Aaron Chan wrote:
> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
> The socket module allows the retrieval on FQDN and constructs the entire
> URL by default, this default settings can be overwritten in c['buildbotURL']
> based on local administrator preferences.
>
> Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
> ---
>   master.cfg | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/master.cfg b/master.cfg
> index fca80d2..49ddeb4 100644
> --- a/master.cfg
> +++ b/master.cfg
> @@ -4,6 +4,7 @@
>   import os
>   import imp
>   import pkg_resources
> +import socket
>
>   from buildbot.plugins import *
>   from buildbot.plugins import db
> @@ -55,6 +56,7 @@ imp.reload(services)
>   imp.reload(www)
>
>   c = BuildmasterConfig = {}
> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port) + '/')

Why use `os.path.join()` here? It isn't supposed to be used to construct
url's, and is overkill for this case, and you'd end up with "http:///...".

// Martin

>
>   # Disable usage reporting
>   c['buildbotNetUsageData'] = None
> @@ -76,6 +78,7 @@ c['www'] = www.www
>   c['workers'] = workers.workers
>
>   c['title'] = "Yocto Autobuilder"
> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
> +c['titleURL'] = url
>   # visible location for internal web server
> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
> +c['buildbotURL'] = url
>

--
Kind regards,
Martin Hundebøll
Embedded Linux Consultant

+45 61 65 54 61
martin@geanix.com

Geanix IVS
DK39600706

Re: [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

[Martin Hundebøll]

Hi Aaron,

On 2018-07-10 08:59, Chan, Aaron Chun Yew wrote:
> My name is Aaron and not Aron for start

Sorry about that.

> Martin,
> 
> Please try this
> 
> #!/usr/bin/env python2
> 
> import os
> 
> a = os.path.join('http://', 'alibaba.com')
> b = '/'.join(['http://', 'alibaba.com'])
> c = '/'.join(['http:/', 'alibaba.com', ''])
> 
> print(a)
> print(b)
> print(c)
> 
> and repeat the same for python3, I got the following results:
> 
> http://alibaba.com
> http:///alibaba.com
> http://alibaba.com/

I see, thanks for clarifying.

My initial concern was that `os.path.join()` was meant for OS 
independent path concatenation, and not constructing URLs.

My two first points still stand, though... Why not just 
string-concatenate the separate parts; i.e.

url = 'http://' + socket.getfqdn() + ':' + str(config.web_port) + '/'

But I won't stand in the way of the patch for such a stylish point, so 
feel free to ignore it :)

// Martin

> Cheers,
> Aaron
> ________________________________________
> From: Martin Hundebøll [martin@geanix.com]
> Sent: Tuesday, July 10, 2018 2:10 PM
> To: Chan, Aaron Chun Yew; richard.purdie@linuxfoundation.org; yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
> Subject: Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN
> 
> Hi Aron,
> 
> On 2018-07-10 05:18, Aaron Chan wrote:
>> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
>> The socket module allows the retrieval on FQDN and constructs the entire
>> URL by default, this default settings can be overwritten in c['buildbotURL']
>> based on local administrator preferences.
>>
>> Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
>> ---
>>    master.cfg | 7 +++++--
>>    1 file changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/master.cfg b/master.cfg
>> index fca80d2..49ddeb4 100644
>> --- a/master.cfg
>> +++ b/master.cfg
>> @@ -4,6 +4,7 @@
>>    import os
>>    import imp
>>    import pkg_resources
>> +import socket
>>
>>    from buildbot.plugins import *
>>    from buildbot.plugins import db
>> @@ -55,6 +56,7 @@ imp.reload(services)
>>    imp.reload(www)
>>
>>    c = BuildmasterConfig = {}
>> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port) + '/')
> 
> Why use `os.path.join()` here? It isn't supposed to be used to construct
> url's, and is overkill for this case, and you'd end up with "http:///...".
> 
> // Martin
> 
>>
>>    # Disable usage reporting
>>    c['buildbotNetUsageData'] = None
>> @@ -76,6 +78,7 @@ c['www'] = www.www
>>    c['workers'] = workers.workers
>>
>>    c['title'] = "Yocto Autobuilder"
>> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
>> +c['titleURL'] = url
>>    # visible location for internal web server
>> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
>> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
>> +c['buildbotURL'] = url
>>
> 
> --
> Kind regards,
> Martin Hundebøll
> Embedded Linux Consultant
> 
> +45 61 65 54 61
> martin@geanix.com
> 
> Geanix IVS
> DK39600706
> 

-- 
Kind regards,
Martin Hundebøll
Embedded Linux Consultant

+45 61 65 54 61
martin@geanix.com

Geanix IVS
DK39600706

Re: [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

[Chan, Aaron Chun Yew]

Hi Martin,

My initial concern was that `os.path.join()` was meant for OS
independent path concatenation, and not constructing URLs.

My two first points still stand, though... Why not just
string-concatenate the separate parts; i.e.

url = 'http://' + socket.getfqdn() + ':' + str(config.web_port) + '/'

[Reply] You have a valid point, however in this case seems like os.path.join() doesn't seem to construct a path. It's a bug. 
            Will send a new patch on this one. Thanks for reviewing it.

Cheers,
Aaron
________________________________________
From: Martin Hundebøll [martin@geanix.com]
Sent: Tuesday, July 10, 2018 3:14 PM
To: Chan, Aaron Chun Yew; richard.purdie@linuxfoundation.org; yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
Subject: Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

Hi Aaron,

On 2018-07-10 08:59, Chan, Aaron Chun Yew wrote:
> My name is Aaron and not Aron for start

Sorry about that.

> Martin,
>
> Please try this
>
> #!/usr/bin/env python2
>
> import os
>
> a = os.path.join('http://', 'alibaba.com')
> b = '/'.join(['http://', 'alibaba.com'])
> c = '/'.join(['http:/', 'alibaba.com', ''])
>
> print(a)
> print(b)
> print(c)
>
> and repeat the same for python3, I got the following results:
>
> http://alibaba.com
> http:///alibaba.com
> http://alibaba.com/

I see, thanks for clarifying.

My initial concern was that `os.path.join()` was meant for OS
independent path concatenation, and not constructing URLs.

My two first points still stand, though... Why not just
string-concatenate the separate parts; i.e.

url = 'http://' + socket.getfqdn() + ':' + str(config.web_port) + '/'

But I won't stand in the way of the patch for such a stylish point, so
feel free to ignore it :)

// Martin

> Cheers,
> Aaron
> ________________________________________
> From: Martin Hundebøll [martin@geanix.com]
> Sent: Tuesday, July 10, 2018 2:10 PM
> To: Chan, Aaron Chun Yew; richard.purdie@linuxfoundation.org; yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
> Subject: Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN
>
> Hi Aron,
>
> On 2018-07-10 05:18, Aaron Chan wrote:
>> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
>> The socket module allows the retrieval on FQDN and constructs the entire
>> URL by default, this default settings can be overwritten in c['buildbotURL']
>> based on local administrator preferences.
>>
>> Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
>> ---
>>    master.cfg | 7 +++++--
>>    1 file changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/master.cfg b/master.cfg
>> index fca80d2..49ddeb4 100644
>> --- a/master.cfg
>> +++ b/master.cfg
>> @@ -4,6 +4,7 @@
>>    import os
>>    import imp
>>    import pkg_resources
>> +import socket
>>
>>    from buildbot.plugins import *
>>    from buildbot.plugins import db
>> @@ -55,6 +56,7 @@ imp.reload(services)
>>    imp.reload(www)
>>
>>    c = BuildmasterConfig = {}
>> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port) + '/')
>
> Why use `os.path.join()` here? It isn't supposed to be used to construct
> url's, and is overkill for this case, and you'd end up with "http:///...".
>
> // Martin
>
>>
>>    # Disable usage reporting
>>    c['buildbotNetUsageData'] = None
>> @@ -76,6 +78,7 @@ c['www'] = www.www
>>    c['workers'] = workers.workers
>>
>>    c['title'] = "Yocto Autobuilder"
>> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
>> +c['titleURL'] = url
>>    # visible location for internal web server
>> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
>> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
>> +c['buildbotURL'] = url
>>
>
> --
> Kind regards,
> Martin Hundebøll
> Embedded Linux Consultant
>
> +45 61 65 54 61
> martin@geanix.com
>
> Geanix IVS
> DK39600706
>

--
Kind regards,
Martin Hundebøll
Embedded Linux Consultant

+45 61 65 54 61
martin@geanix.com

Geanix IVS
DK39600706

Re: [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

[Richard Purdie]

On Tue, 2018-07-10 at 11:18 +0800, Aaron Chan wrote:
> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
> The socket module allows the retrieval on FQDN and constructs the entire
> URL by default, this default settings can be overwritten in c['buildbotURL']
> based on local administrator preferences.
> 
> Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
> ---
>  master.cfg | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/master.cfg b/master.cfg
> index fca80d2..49ddeb4 100644
> --- a/master.cfg
> +++ b/master.cfg
> @@ -4,6 +4,7 @@
>  import os
>  import imp
>  import pkg_resources
> +import socket
>  
>  from buildbot.plugins import *
>  from buildbot.plugins import db
> @@ -55,6 +56,7 @@ imp.reload(services)
>  imp.reload(www)
>  
>  c = BuildmasterConfig = {}
> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port) + '/')
>  
>  # Disable usage reporting
>  c['buildbotNetUsageData'] = None
> @@ -76,6 +78,7 @@ c['www'] = www.www
>  c['workers'] = workers.workers
>  
>  c['title'] = "Yocto Autobuilder"
> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
> +c['titleURL'] = url
>  # visible location for internal web server
> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
> +c['buildbotURL'] = url

I appreciate what you're trying to do here and make this autoconfigure.
 Unfortunately the urls can't always be figured out this way, the above
for example drops the "/main/" part, without which the autobuilder
won't work. The server can be behind forwarding or proxies/firewalls
which also make this problematic.

I also agree with Martin that using os.path.join() in a url is
incorrect, its not meant for urls.

Most people setting up an autobuilder will need to have some
customisations on top of yocto-autobuilder2, I don't think its possible
to avoid that. I'd therefore perhaps try and concentrate on having key
modules like the lava integration available and accept there will
always be some local configuration the end user needs to make to things
like the URL details? Even the main autobuilder adds in passwords and
some other local config on top of the stardard repo...

Cheers,

Richard

Re: [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

[Chan, Aaron Chun Yew]

Hi Richard,

[Richard] I appreciate what you're trying to do here and make this autoconfigure.
 Unfortunately the urls can't always be figured out this way, the above
for example drops the "/main/" part, without which the autobuilder
won't work. The server can be behind forwarding or proxies/firewalls
which also make this problematic.

I also agree with Martin that using os.path.join() in a url is
incorrect, its not meant for urls.

Most people setting up an autobuilder will need to have some
customisations on top of yocto-autobuilder2, I don't think its possible
to avoid that. I'd therefore perhaps try and concentrate on having key
modules like the lava integration available and accept there will
always be some local configuration the end user needs to make to things
like the URL details? Even the main autobuilder adds in passwords and
some other local config on top of the stardard repo...

[Reply] I do agree with you that some customization are required to be built on top of autobuilder.
            However this patch is submitted because the URL link below is not working on my end and
            only http://localhost:8010/ is working without the need to change the master.cfg
            So therefore, due to several rounds of commission i decided to send a patch which defaults
            URL following the FQDN sets on the host machines.

            c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
            
            c['buildbotURL'] = "http://localhost:8010/'

The intention of this patch is to reduce the need for local configurations, yes I do agree that there will
be some level of customization needed on local setup. I'll leave it to you then to determine whats best.

Cheers,
Aaron

________________________________________
From: richard.purdie@linuxfoundation.org [richard.purdie@linuxfoundation.org]
Sent: Tuesday, July 10, 2018 4:55 PM
To: Chan, Aaron Chun Yew; yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
Subject: Re: [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN

On Tue, 2018-07-10 at 11:18 +0800, Aaron Chan wrote:
> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
> The socket module allows the retrieval on FQDN and constructs the entire
> URL by default, this default settings can be overwritten in c['buildbotURL']
> based on local administrator preferences.
>
> Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
> ---
>  master.cfg | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/master.cfg b/master.cfg
> index fca80d2..49ddeb4 100644
> --- a/master.cfg
> +++ b/master.cfg
> @@ -4,6 +4,7 @@
>  import os
>  import imp
>  import pkg_resources
> +import socket
>
>  from buildbot.plugins import *
>  from buildbot.plugins import db
> @@ -55,6 +56,7 @@ imp.reload(services)
>  imp.reload(www)
>
>  c = BuildmasterConfig = {}
> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port) + '/')
>
>  # Disable usage reporting
>  c['buildbotNetUsageData'] = None
> @@ -76,6 +78,7 @@ c['www'] = www.www
>  c['workers'] = workers.workers
>
>  c['title'] = "Yocto Autobuilder"
> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
> +c['titleURL'] = url
>  # visible location for internal web server
> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/"
> +c['buildbotURL'] = url

I appreciate what you're trying to do here and make this autoconfigure.
 Unfortunately the urls can't always be figured out this way, the above
for example drops the "/main/" part, without which the autobuilder
won't work. The server can be behind forwarding or proxies/firewalls
which also make this problematic.

I also agree with Martin that using os.path.join() in a url is
incorrect, its not meant for urls.

Most people setting up an autobuilder will need to have some
customisations on top of yocto-autobuilder2, I don't think its possible
to avoid that. I'd therefore perhaps try and concentrate on having key
modules like the lava integration available and accept there will
always be some local configuration the end user needs to make to things
like the URL details? Even the main autobuilder adds in passwords and
some other local config on top of the stardard repo...

Cheers,

Richard