[Buildroot] [PATCH 01/14] support/scripts/pkg-stats: URL checking support

[Buildroot] [PATCH 01/14] support/scripts/pkg-stats: URL checking support

[Matt Weber]

 - Adds support to check if a package has a URL and if that URL
   is valid by doing a header request.
 - Reports this information as part of the generated html output

This check helps ensure the URLs are valid and can be used
for other scripting purposes as the product's home site/URL.
CPE XML generation is an example of a case that could use this
product URL as part of an automated update generation script.

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 support/scripts/pkg-stats | 37 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index b7b00e8..c83545b 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -24,6 +24,8 @@ from collections import defaultdict
 import re
 import subprocess
 import sys
+import time
+import requests  # URL checking
 
 INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
 
@@ -43,10 +45,30 @@ class Package:
         self.patch_count = 0
         self.warnings = 0
         self.current_version = None
+        self.url = None
 
     def pkgvar(self):
         return self.name.upper().replace("-", "_")
 
+    def set_url(self):
+        """
+        Fills in the .url field
+        """
+        in_help_section = False
+        self.url = "No Config"
+        for filename in os.listdir(os.path.dirname(self.path)):
+            if fnmatch.fnmatch(filename, 'Config.*'):
+                fp = open(os.path.join(os.path.dirname(self.path), filename), "r")
+                for config_line in fp:
+                    if config_line.strip() == "help":
+                        in_help_section = True
+                    if in_help_section and re.match("(.*)https?://", config_line):
+                        self.url = ''.join(config_line.split())
+                        fp.close()
+                        return
+                self.url = "Missing Entry"
+                fp.close()
+
     def set_infra(self):
         """
         Fills in the .infras field
@@ -356,7 +378,19 @@ def boolean_str(b):
 
 def dump_html_pkg(f, pkg):
     f.write(" <tr>\n")
-    f.write("  <td>%s</td>\n" % pkg.path[2:])
+    url_status = "Ok"
+    if str(pkg.url) == "Missing Entry":
+        f.write("  <td>%s<br>    (URL: Missing URL)</td>\n" % pkg.path[2:])
+    elif str(pkg.url) == "No Config":
+        f.write("  <td>%s<br>    (URL: No Config File)</td>\n" % pkg.path[2:])
+    else:
+        try:
+            url_status_code = requests.head(pkg.url, verify=False).status_code
+            if url_status_code > 308:
+                url_status = "Error(" + str(url_status_code) + ")"
+        except requests.exceptions.RequestException as e:
+            url_status = e
+        f.write("  <td>%s<br>    (<a href=%s>URL: %s</a>)</td>\n" % (pkg.path[2:], str(pkg.url), url_status))
 
     # Patch count
     td_class = ["centered"]
@@ -511,6 +545,7 @@ def __main__():
     package_init_make_info()
     print("Getting package details ...")
     for pkg in packages:
+        pkg.set_url()
         pkg.set_infra()
         pkg.set_license()
         pkg.set_hash_info()
-- 
1.9.1

[Buildroot] [PATCH 02/14] boot/at91bootstrap: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 boot/at91bootstrap/Config.in | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/boot/at91bootstrap/Config.in b/boot/at91bootstrap/Config.in
index 0f6430f..3d649b1 100644
--- a/boot/at91bootstrap/Config.in
+++ b/boot/at91bootstrap/Config.in
@@ -10,6 +10,8 @@ config BR2_TARGET_AT91BOOTSTRAP
 	  - Physical media algorithm such as DataFlash, NandFlash, NOR
 	    Flash...
 
+	  https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
+
 if	BR2_TARGET_AT91BOOTSTRAP
 
 config BR2_TARGET_AT91BOOTSTRAP_CUSTOM_PATCH_DIR
-- 
1.9.1

[Buildroot] [PATCH 03/14] boot/at91bootstrap3: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 boot/at91bootstrap3/Config.in | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/boot/at91bootstrap3/Config.in b/boot/at91bootstrap3/Config.in
index 614363c..4534b4e 100644
--- a/boot/at91bootstrap3/Config.in
+++ b/boot/at91bootstrap3/Config.in
@@ -10,6 +10,9 @@ config BR2_TARGET_AT91BOOTSTRAP3
 	  - Physical media algorithm such as DataFlash, NandFlash, NOR
 	    Flash...
 
+
+	  https://www.at91.com/linux4sam/bin/view/Linux4SAM/AT91Bootstrap
+
 if BR2_TARGET_AT91BOOTSTRAP3
 
 choice
-- 
1.9.1

[Buildroot] [PATCH 04/14] boot/at91dataflashboot: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 boot/at91dataflashboot/Config.in | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/boot/at91dataflashboot/Config.in b/boot/at91dataflashboot/Config.in
index 4055f96..1a95709 100644
--- a/boot/at91dataflashboot/Config.in
+++ b/boot/at91dataflashboot/Config.in
@@ -1,3 +1,5 @@
 config BR2_TARGET_AT91DATAFLASHBOOT
 	bool "AT91 DataFlashBoot"
 	depends on BR2_arm926t
+	help
+	  https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
-- 
1.9.1

[Buildroot] [PATCH 05/14] boot/xloader: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 boot/xloader/Config.in | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/boot/xloader/Config.in b/boot/xloader/Config.in
index c411be4..f687b89 100644
--- a/boot/xloader/Config.in
+++ b/boot/xloader/Config.in
@@ -5,6 +5,8 @@ config BR2_TARGET_XLOADER
 	  The x-loader bootloader. It is mainly used on OMAP-based
 	  platforms.
 
+	  http://omappedia.org/wiki/Linux_OMAP_Kernel_Main
+
 if BR2_TARGET_XLOADER
 config BR2_TARGET_XLOADER_BOARDNAME
 	string "x-loader board name"
-- 
1.9.1

[Buildroot] [PATCH 06/14] package/android-tools: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 package/android-tools/Config.in | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/android-tools/Config.in b/package/android-tools/Config.in
index 96e36e6..478f139 100644
--- a/package/android-tools/Config.in
+++ b/package/android-tools/Config.in
@@ -15,6 +15,8 @@ config BR2_PACKAGE_ANDROID_TOOLS
 	  can be used to interact with target devices using of these
 	  protocols.
 
+	  https://wiki.debian.org/AndroidTools#Original_android-tools_package
+
 if BR2_PACKAGE_ANDROID_TOOLS
 
 # We need kernel headers that support the __SANE_USERSPACE_TYPES__
-- 
1.9.1

[Buildroot] [PATCH 07/14] package/arp-scan: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 package/arp-scan/Config.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/arp-scan/Config.in b/package/arp-scan/Config.in
index cc47a72..ed70b39 100644
--- a/package/arp-scan/Config.in
+++ b/package/arp-scan/Config.in
@@ -7,4 +7,4 @@ config BR2_PACKAGE_ARP_SCAN
 	  arp-scan is a command-line tool that uses the ARP protocol to
 	  discover and fingerprint IP hosts on the local network.
 
-	  http://www.nta-monitor.com/wiki/index.php/Arp-scan_Documentation
+	  https://github.com/royhills/arp-scan
-- 
1.9.1

[Buildroot] [PATCH 08/14] package/bandwidthd: URL move to newline

[Matt Weber]

Allows scripting of URL checking to be simplier

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 package/bandwidthd/Config.in | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/package/bandwidthd/Config.in b/package/bandwidthd/Config.in
index a503da4..8a19733 100644
--- a/package/bandwidthd/Config.in
+++ b/package/bandwidthd/Config.in
@@ -25,8 +25,10 @@ config BR2_PACKAGE_BANDWIDTHD
 	  available on github that works on making BandwidthD's build
 	  process more compatible with buildroot's.
 
-	  Upstream: http://bandwidthd.sourceforge.net/
-	  Github fork: http://github.com/nroach44/bandwidthd
+	  Upstream:
+	  http://bandwidthd.sourceforge.net/
+	  Github fork:
+	  http://github.com/nroach44/bandwidthd
 
 if BR2_PACKAGE_BANDWIDTHD
 
-- 
1.9.1

[Buildroot] [PATCH 09/14] package/connman: URL move to newline

[Matt Weber]

Allows scripting of URL checking to be simplier

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 package/connman/Config.in | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/connman/Config.in b/package/connman/Config.in
index 253c604..0f93fee 100644
--- a/package/connman/Config.in
+++ b/package/connman/Config.in
@@ -14,7 +14,8 @@ config BR2_PACKAGE_CONNMAN
 	  for managing internet connections within embedded devices
 	  running the Linux operating system.
 
-	  For more information, see https://01.org/connman
+	  For more information, see
+	  https://01.org/connman
 
 if BR2_PACKAGE_CONNMAN
 
-- 
1.9.1

[Buildroot] [PATCH 10/14] package/glibc: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 package/glibc/Config.in | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/glibc/Config.in b/package/glibc/Config.in
index 57a2e83..63dd0f4 100644
--- a/package/glibc/Config.in
+++ b/package/glibc/Config.in
@@ -5,5 +5,6 @@ config BR2_PACKAGE_GLIBC
 	default y
 	select BR2_PACKAGE_LINUX_HEADERS
 	select BR2_TOOLCHAIN_HAS_SSP
-
+	help
+	  https://www.gnu.org/software/libc/
 endif
-- 
1.9.1

[Buildroot] [PATCH 11/14] package/tidsp-binaries: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 package/tidsp-binaries/Config.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/tidsp-binaries/Config.in b/package/tidsp-binaries/Config.in
index 48f85a5..24b2891 100644
--- a/package/tidsp-binaries/Config.in
+++ b/package/tidsp-binaries/Config.in
@@ -4,4 +4,4 @@ config BR2_PACKAGE_TIDSP_BINARIES
 	help
 	  TI OMAP3 DSP algorithms.
 
-	  https://gforge.ti.com/gf/project/openmax/
+	  http://omappedia.org/wiki/OpenMAX_Project
-- 
1.9.1

[Buildroot] [PATCH 12/14] boot/vexpress-firmware: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 boot/vexpress-firmware/Config.in | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/boot/vexpress-firmware/Config.in b/boot/vexpress-firmware/Config.in
index 8479df3..265fb4b 100644
--- a/boot/vexpress-firmware/Config.in
+++ b/boot/vexpress-firmware/Config.in
@@ -4,3 +4,5 @@ config BR2_TARGET_VEXPRESS_FIRMWARE
 	help
 	  Versatile Express firmware from ARM, with Linaro mods last
 	  change.
+
+	  https://git.linaro.org/arm/vexpress-firmware.git
-- 
1.9.1

[Buildroot] [PATCH 13/14] package/alljoyn-base: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 package/alljoyn-base/Config.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/alljoyn-base/Config.in b/package/alljoyn-base/Config.in
index 8556d66..c89cb20 100644
--- a/package/alljoyn-base/Config.in
+++ b/package/alljoyn-base/Config.in
@@ -17,7 +17,7 @@ config BR2_PACKAGE_ALLJOYN_BASE
 	  devices, providing a set of interfaces for different devices
 	  to interact and interoperate with one another.
 
-	  https://allseenalliance.org
+	  https://github.com/alljoyn/alljoyn.github.com/wiki
 
 if BR2_PACKAGE_ALLJOYN_BASE
 
-- 
1.9.1

[Buildroot] [PATCH 14/14] package/alljoyn-tcl-base: URL update

[Matt Weber]

Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
---
 package/alljoyn-tcl-base/Config.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/alljoyn-tcl-base/Config.in b/package/alljoyn-tcl-base/Config.in
index ebc3370..404957a 100644
--- a/package/alljoyn-tcl-base/Config.in
+++ b/package/alljoyn-tcl-base/Config.in
@@ -12,7 +12,7 @@ config BR2_PACKAGE_ALLJOYN_TCL_BASE
 	  AllJoyn distributed programming environment to embedded
 	  systems.
 
-	  https://allseenalliance.org
+	  https://github.com/alljoyn/alljoyn.github.com/wiki
 
 comment "alljoyn-tcl-base needs a toolchain w/ threads and dynamic library"
 	depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_STATIC_LIBS
-- 
1.9.1

[Buildroot] [PATCH 09/14] package/connman: URL move to newline

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:54 -0500, Matt Weber wrote:
> Allows scripting of URL checking to be simplier
> 
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  package/connman/Config.in | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/package/connman/Config.in b/package/connman/Config.in
> index 253c604..0f93fee 100644
> --- a/package/connman/Config.in
> +++ b/package/connman/Config.in
> @@ -14,7 +14,8 @@ config BR2_PACKAGE_CONNMAN
>  	  for managing internet connections within embedded devices
>  	  running the Linux operating system.
>  
> -	  For more information, see https://01.org/connman
> +	  For more information, see
> +	  https://01.org/connman

Actually, the "For more information, see" is totally useless, so I've
dropped that instead to solve the problem of "we want the URL at the
beginning of a line".

Applied with this change. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 10/14] package/glibc: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:55 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  package/glibc/Config.in | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/package/glibc/Config.in b/package/glibc/Config.in
> index 57a2e83..63dd0f4 100644
> --- a/package/glibc/Config.in
> +++ b/package/glibc/Config.in
> @@ -5,5 +5,6 @@ config BR2_PACKAGE_GLIBC
>  	default y
>  	select BR2_PACKAGE_LINUX_HEADERS
>  	select BR2_TOOLCHAIN_HAS_SSP
> -
> +	help
> +	  https://www.gnu.org/software/libc/

This option is a blind option, so having a help text is not very
useful...

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 11/14] package/tidsp-binaries: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:56 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  package/tidsp-binaries/Config.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/tidsp-binaries/Config.in b/package/tidsp-binaries/Config.in
> index 48f85a5..24b2891 100644
> --- a/package/tidsp-binaries/Config.in
> +++ b/package/tidsp-binaries/Config.in
> @@ -4,4 +4,4 @@ config BR2_PACKAGE_TIDSP_BINARIES
>  	help
>  	  TI OMAP3 DSP algorithms.
>  
> -	  https://gforge.ti.com/gf/project/openmax/
> +	  http://omappedia.org/wiki/OpenMAX_Project

Is that really the right URL ? It points to gforge.ti.com, which indeed
seems to be down. Should we simply drop this package perhaps ? The
upstream site is not available, I doubt anybody is using it: it was
introduced in 2011 and never touched again in any useful/meaningful way.

So I would propose to drop this package.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 12/14] boot/vexpress-firmware: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:57 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  boot/vexpress-firmware/Config.in | 2 ++
>  1 file changed, 2 insertions(+)

The title says "URL update", but that's not true: this commit is adding
the URL. A better commit title:

  boot/vexpress-firmware: add upstream URL in Config.in help text

I fixed that and applied. Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 13/14] package/alljoyn-base: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:58 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  package/alljoyn-base/Config.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/alljoyn-base/Config.in b/package/alljoyn-base/Config.in
> index 8556d66..c89cb20 100644
> --- a/package/alljoyn-base/Config.in
> +++ b/package/alljoyn-base/Config.in
> @@ -17,7 +17,7 @@ config BR2_PACKAGE_ALLJOYN_BASE
>  	  devices, providing a set of interfaces for different devices
>  	  to interact and interoperate with one another.
>  
> -	  https://allseenalliance.org
> +	  https://github.com/alljoyn/alljoyn.github.com/wiki

For PATCH 13/14 (this one) and PATCH 14/14, I'd like something a bit
better:

 - The .mk files still use a mirror of allseenalliance at kernel.org,
   it should be changed to use the Github location instead.

 - You changed alljoyn-base and alljoyn-tcl-base, but not alljoyn-tcl

 - The commit title should be "update upstream URL"

 - The commit log should contain some details, at least explaining that
   allseenalliance.org is dead, and that the project was moved to
   Github, etc.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 10/14] package/glibc: URL update

[Matthew Weber]

Thomas,
On Thu, Sep 20, 2018 at 2:30 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Thu, 20 Sep 2018 08:24:55 -0500, Matt Weber wrote:
> > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> > ---
> >  package/glibc/Config.in | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/package/glibc/Config.in b/package/glibc/Config.in
> > index 57a2e83..63dd0f4 100644
> > --- a/package/glibc/Config.in
> > +++ b/package/glibc/Config.in
> > @@ -5,5 +5,6 @@ config BR2_PACKAGE_GLIBC
> >       default y
> >       select BR2_PACKAGE_LINUX_HEADERS
> >       select BR2_TOOLCHAIN_HAS_SSP
> > -
> > +     help
> > +       https://www.gnu.org/software/libc/
>
> This option is a blind option, so having a help text is not very
> useful...

I had been looking at verifying and updating all packages which could
end-up with content on target to have a valid URL.  This allows
auto-generation of CPE XML update requests which need product site
urls.

Matt

[Buildroot] [PATCH 03/14] boot/at91bootstrap3: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:48 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  boot/at91bootstrap3/Config.in | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/boot/at91bootstrap3/Config.in b/boot/at91bootstrap3/Config.in
> index 614363c..4534b4e 100644
> --- a/boot/at91bootstrap3/Config.in
> +++ b/boot/at91bootstrap3/Config.in
> @@ -10,6 +10,9 @@ config BR2_TARGET_AT91BOOTSTRAP3
>  	  - Physical media algorithm such as DataFlash, NandFlash, NOR
>  	    Flash...
>  
> +

One too many newline.

> +	  https://www.at91.com/linux4sam/bin/view/Linux4SAM/AT91Bootstrap

And the commit title was wrong: you're not updating the URL, but adding
it.

Applied after fixing those issues.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 02/14] boot/at91bootstrap: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:47 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  boot/at91bootstrap/Config.in | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/boot/at91bootstrap/Config.in b/boot/at91bootstrap/Config.in
> index 0f6430f..3d649b1 100644
> --- a/boot/at91bootstrap/Config.in
> +++ b/boot/at91bootstrap/Config.in
> @@ -10,6 +10,8 @@ config BR2_TARGET_AT91BOOTSTRAP
>  	  - Physical media algorithm such as DataFlash, NandFlash, NOR
>  	    Flash...
>  
> +	  https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap

Actually, this one should use:

	  https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap

which you have used for at91dataflashboot, but at91dataflashboot is a
different thing, which no longer has any upstream location.

Perhaps your script should support a way of specifying "yeah, there is
no upstream location" in an explicit way.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 10/14] package/glibc: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 15:05:46 -0500, Matthew Weber wrote:

> > This option is a blind option, so having a help text is not very
> > useful...  
> 
> I had been looking at verifying and updating all packages which could
> end-up with content on target to have a valid URL.  This allows
> auto-generation of CPE XML update requests which need product site
> urls.

Then perhaps this should be explained in the commit log, specifically
for glibc.

But are you sure your approach is the right one ? Today only target
packages have a Config.in file. What are you going to do for host
packages ? You don't care because they are host packages and supposedly
not relevant for security issues on the target ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 06/14] package/android-tools: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:51 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  package/android-tools/Config.in | 2 ++
>  1 file changed, 2 insertions(+)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 07/14] package/arp-scan: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:52 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  package/arp-scan/Config.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 08/14] package/bandwidthd: URL move to newline

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:53 -0500, Matt Weber wrote:
> Allows scripting of URL checking to be simplier
> 
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  package/bandwidthd/Config.in | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 05/14] boot/xloader: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 08:24:50 -0500, Matt Weber wrote:
> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> ---
>  boot/xloader/Config.in | 2 ++
>  1 file changed, 2 insertions(+)

I've fixed the commit title: you're adding an upstream URL, not
updating it, and I've applied.

However:
 
 - The link isn't really a useful upstream URL. There in fact doesn't
   seem to be any page that is really relevant for X-Loader.

 - I am not sure having a package for xloader still makes sense. It's
   completely deprecated now that U-Boot has SPL support for TI
   platforms. Perhaps we should simply remove it ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 10/14] package/glibc: URL update

[Matthew Weber]

Thomas,

On Thu, Sep 20, 2018 at 3:17 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Thu, 20 Sep 2018 15:05:46 -0500, Matthew Weber wrote:
>
> > > This option is a blind option, so having a help text is not very
> > > useful...
> >
> > I had been looking at verifying and updating all packages which could
> > end-up with content on target to have a valid URL.  This allows
> > auto-generation of CPE XML update requests which need product site
> > urls.
>
> Then perhaps this should be explained in the commit log, specifically
> for glibc.
>
Yes, agree.

> But are you sure your approach is the right one ? Today only target
> packages have a Config.in file. What are you going to do for host
> packages ?
Currently the xml generation was a best effort and the cpe reporting
was only done against target CPE so far.....  Glad to discuss
suggestions for a better way vs hand updating the xml with this URL
info.

> You don't care because they are host packages and supposedly
> not relevant for security issues on the target ?

I know better then to say that host packages don't matter :-)  We took
the most accessible path first and that was to get good coverage on
target packages.

Maybe a good topic for next month?  I should have an updated CPE
patchset out in the next week.

Matt

[Buildroot] [PATCH 02/14] boot/at91bootstrap: URL update

[Matthew Weber]

On Thu, Sep 20, 2018 at 3:13 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Thu, 20 Sep 2018 08:24:47 -0500, Matt Weber wrote:
> > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> > ---
> >  boot/at91bootstrap/Config.in | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/boot/at91bootstrap/Config.in b/boot/at91bootstrap/Config.in
> > index 0f6430f..3d649b1 100644
> > --- a/boot/at91bootstrap/Config.in
> > +++ b/boot/at91bootstrap/Config.in
> > @@ -10,6 +10,8 @@ config BR2_TARGET_AT91BOOTSTRAP
> >         - Physical media algorithm such as DataFlash, NandFlash, NOR
> >           Flash...
> >
> > +       https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
>
> Actually, this one should use:
>
>           https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
>

Ok so this patch is ok then and for the at91dataflashboot I found it
mentioned on the same legacy page but it definitely is a weak
reference.


> which you have used for at91dataflashboot, but at91dataflashboot is a
> different thing, which no longer has any upstream location.
>
> Perhaps your script should support a way of specifying "yeah, there is
> no upstream location" in an explicit way.

Is that maybe a string we put in the end of the help section?   "No
Upstream URL"

Matt

[Buildroot] [PATCH 13/14] package/alljoyn-base: URL update

[Matthew Weber]

Thomas/Fabrice,

On Thu, Sep 20, 2018 at 3:05 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Thu, 20 Sep 2018 08:24:58 -0500, Matt Weber wrote:
> > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> > ---
> >  package/alljoyn-base/Config.in | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/package/alljoyn-base/Config.in b/package/alljoyn-base/Config.in
> > index 8556d66..c89cb20 100644
> > --- a/package/alljoyn-base/Config.in
> > +++ b/package/alljoyn-base/Config.in
> > @@ -17,7 +17,7 @@ config BR2_PACKAGE_ALLJOYN_BASE
> >         devices, providing a set of interfaces for different devices
> >         to interact and interoperate with one another.
> >
> > -       https://allseenalliance.org
> > +       https://github.com/alljoyn/alljoyn.github.com/wiki
>
> For PATCH 13/14 (this one) and PATCH 14/14, I'd like something a bit
> better:
>
>  - The .mk files still use a mirror of allseenalliance at kernel.org,
>    it should be changed to use the Github location instead.

I don't have a good use case to runtime test making this change.  I
can make it an verified the build is still good.

>
>  - You changed alljoyn-base and alljoyn-tcl-base, but not alljoyn-tcl

I'll have to check, my script may have a bug.  My grep is coming back
showing I missed the plain alljoyn too.
alljoyn-base        https://github.com/alljoyn/alljoyn.github.com/wiki
alljoyn                https://allseenalliance.org
alljoyn-tcl-base    https://github.com/alljoyn/alljoyn.github.com/wiki
alljoyn-tcl            https://allseenalliance.org

>
>  - The commit title should be "update upstream URL"

Noted.

>
>  - The commit log should contain some details, at least explaining that
>    allseenalliance.org is dead, and that the project was moved to
>    Github, etc.

Sure, I'm not to familiar with this one but I can at least state that.

Matt

[Buildroot] [PATCH 01/14] support/scripts/pkg-stats: URL checking support

[Thomas Petazzoni]

Hello Matt,

On Thu, 20 Sep 2018 08:24:46 -0500, Matt Weber wrote:

> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index b7b00e8..c83545b 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -24,6 +24,8 @@ from collections import defaultdict
>  import re
>  import subprocess
>  import sys
> +import time
> +import requests  # URL checking
>  
>  INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
>  
> @@ -43,10 +45,30 @@ class Package:
>          self.patch_count = 0
>          self.warnings = 0
>          self.current_version = None
> +        self.url = None
>  
>      def pkgvar(self):
>          return self.name.upper().replace("-", "_")
>  
> +    def set_url(self):
> +        """
> +        Fills in the .url field
> +        """
> +        in_help_section = False
> +        self.url = "No Config"
> +        for filename in os.listdir(os.path.dirname(self.path)):
> +            if fnmatch.fnmatch(filename, 'Config.*'):
> +                fp = open(os.path.join(os.path.dirname(self.path), filename), "r")
> +                for config_line in fp:
> +                    if config_line.strip() == "help":
> +                        in_help_section = True
> +                    if in_help_section and re.match("(.*)https?://", config_line):
> +                        self.url = ''.join(config_line.split())
> +                        fp.close()
> +                        return
> +                self.url = "Missing Entry"
> +                fp.close()
> +
>      def set_infra(self):
>          """
>          Fills in the .infras field
> @@ -356,7 +378,19 @@ def boolean_str(b):
>  
>  def dump_html_pkg(f, pkg):
>      f.write(" <tr>\n")
> -    f.write("  <td>%s</td>\n" % pkg.path[2:])
> +    url_status = "Ok"
> +    if str(pkg.url) == "Missing Entry":
> +        f.write("  <td>%s<br>    (URL: Missing URL)</td>\n" % pkg.path[2:])
> +    elif str(pkg.url) == "No Config":
> +        f.write("  <td>%s<br>    (URL: No Config File)</td>\n" % pkg.path[2:])
> +    else:
> +        try:
> +            url_status_code = requests.head(pkg.url, verify=False).status_code
> +            if url_status_code > 308:
> +                url_status = "Error(" + str(url_status_code) + ")"
> +        except requests.exceptions.RequestException as e:
> +            url_status = e
> +        f.write("  <td>%s<br>    (<a href=%s>URL: %s</a>)</td>\n" % (pkg.path[2:], str(pkg.url), url_status))

Could you instead add a new column, like we already have for the
version, package type, number of patches, etc. and use the existing
green/orange/red colors to indicate URL present and working (green),
URL not present (orange) and URL present but not working (red) ?

How long does it take to run the script before/after your addition, on
all packages ? I'm sure you remember I was working on adding support
for using release-monitoring.org to this script to keep track of
upstream versions of packages, but doing it sequentially for the 2000+
packages we have was way too slow and I had to use several threads to
speed things up and make it reasonable.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 02/14] boot/at91bootstrap: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 15:35:14 -0500, Matthew Weber wrote:

> > > diff --git a/boot/at91bootstrap/Config.in b/boot/at91bootstrap/Config.in
> > > index 0f6430f..3d649b1 100644
> > > --- a/boot/at91bootstrap/Config.in
> > > +++ b/boot/at91bootstrap/Config.in
> > > @@ -10,6 +10,8 @@ config BR2_TARGET_AT91BOOTSTRAP
> > >         - Physical media algorithm such as DataFlash, NandFlash, NOR
> > >           Flash...
> > >
> > > +       https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap  
> >
> > Actually, this one should use:
> >
> >           https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
> >  
> 
> Ok so this patch is ok then

Ah, yes, my bad, sorry :-/

> and for the at91dataflashboot I found it mentioned on the same legacy
> page but it definitely is a weak reference.

I sent an e-mail to Atmel^WMicrochip people about at91dataflashboot,
we'll see what they have to say.

> > Perhaps your script should support a way of specifying "yeah, there
> > is no upstream location" in an explicit way.  
> 
> Is that maybe a string we put in the end of the help section?   "No
> Upstream URL"

Yes, I was thinking about something like that.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 13/14] package/alljoyn-base: URL update

[Thomas Petazzoni]

Hello,

On Thu, 20 Sep 2018 15:44:54 -0500, Matthew Weber wrote:

> >  - The .mk files still use a mirror of allseenalliance at kernel.org,
> >    it should be changed to use the Github location instead.  
> 
> I don't have a good use case to runtime test making this change.  I
> can make it an verified the build is still good.

I think we should try to keep using the same versions (we use 16.04,
and I saw it was available on Github as well), and preferably try to
diff the source code between the old upstream and the new upstream. If
the code is reasonably similar, then build testing should be sufficient.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 11/14] package/tidsp-binaries: URL update

[Matthew Weber]

Thomas,

On Thu, Sep 20, 2018 at 2:39 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Thu, 20 Sep 2018 08:24:56 -0500, Matt Weber wrote:
> > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
> > ---
> >  package/tidsp-binaries/Config.in | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/package/tidsp-binaries/Config.in b/package/tidsp-binaries/Config.in
> > index 48f85a5..24b2891 100644
> > --- a/package/tidsp-binaries/Config.in
> > +++ b/package/tidsp-binaries/Config.in
> > @@ -4,4 +4,4 @@ config BR2_PACKAGE_TIDSP_BINARIES
> >       help
> >         TI OMAP3 DSP algorithms.
> >
> > -       https://gforge.ti.com/gf/project/openmax/
> > +       http://omappedia.org/wiki/OpenMAX_Project
>
> Is that really the right URL ? It points to gforge.ti.com, which indeed
> seems to be down. Should we simply drop this package perhaps ? The
> upstream site is not available, I doubt anybody is using it: it was
> introduced in 2011 and never touched again in any useful/meaningful way.
>
> So I would propose to drop this package.

This package is used by dsp-tools and gstreamer/gst-dsp so I'd assume
those should also be removed.

This capability feels feature complete when looking at the Gstreamer
stuff....  I guess we'll see if anyone notices the removal.  Should I
have 3 seperate patches for the individual packages?

Matt

[Buildroot] [PATCH 01/14] support/scripts/pkg-stats: URL checking support

[Matthew Weber]

On Thu, Sep 20, 2018 at 3:58 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Matt,
>
> On Thu, 20 Sep 2018 08:24:46 -0500, Matt Weber wrote:
>
> > diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> > index b7b00e8..c83545b 100755
> > --- a/support/scripts/pkg-stats
> > +++ b/support/scripts/pkg-stats
> > @@ -24,6 +24,8 @@ from collections import defaultdict
> >  import re
> >  import subprocess
> >  import sys
> > +import time
> > +import requests  # URL checking
> >
> >  INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
> >
> > @@ -43,10 +45,30 @@ class Package:
> >          self.patch_count = 0
> >          self.warnings = 0
> >          self.current_version = None
> > +        self.url = None
> >
> >      def pkgvar(self):
> >          return self.name.upper().replace("-", "_")
> >
> > +    def set_url(self):
> > +        """
> > +        Fills in the .url field
> > +        """
> > +        in_help_section = False
> > +        self.url = "No Config"
> > +        for filename in os.listdir(os.path.dirname(self.path)):
> > +            if fnmatch.fnmatch(filename, 'Config.*'):
> > +                fp = open(os.path.join(os.path.dirname(self.path), filename), "r")
> > +                for config_line in fp:
> > +                    if config_line.strip() == "help":
> > +                        in_help_section = True
> > +                    if in_help_section and re.match("(.*)https?://", config_line):
> > +                        self.url = ''.join(config_line.split())
> > +                        fp.close()
> > +                        return
> > +                self.url = "Missing Entry"
> > +                fp.close()
> > +
> >      def set_infra(self):
> >          """
> >          Fills in the .infras field
> > @@ -356,7 +378,19 @@ def boolean_str(b):
> >
> >  def dump_html_pkg(f, pkg):
> >      f.write(" <tr>\n")
> > -    f.write("  <td>%s</td>\n" % pkg.path[2:])
> > +    url_status = "Ok"
> > +    if str(pkg.url) == "Missing Entry":
> > +        f.write("  <td>%s<br>    (URL: Missing URL)</td>\n" % pkg.path[2:])
> > +    elif str(pkg.url) == "No Config":
> > +        f.write("  <td>%s<br>    (URL: No Config File)</td>\n" % pkg.path[2:])
> > +    else:
> > +        try:
> > +            url_status_code = requests.head(pkg.url, verify=False).status_code
> > +            if url_status_code > 308:
> > +                url_status = "Error(" + str(url_status_code) + ")"
> > +        except requests.exceptions.RequestException as e:
> > +            url_status = e
> > +        f.write("  <td>%s<br>    (<a href=%s>URL: %s</a>)</td>\n" % (pkg.path[2:], str(pkg.url), url_status))
>
> Could you instead add a new column, like we already have for the
> version, package type, number of patches, etc. and use the existing
> green/orange/red colors to indicate URL present and working (green),
> URL not present (orange) and URL present but not working (red) ?
>

How does this look? (Still some bugs with some of the invalid sites,
the webservers are reporting incorrect codes....)
https://pste.eu/p/JpsH.html

> How long does it take to run the script before/after your addition, on
> all packages ? I'm sure you remember I was working on adding support
> for using release-monitoring.org to this script to keep track of
> upstream versions of packages, but doing it sequentially for the 2000+
> packages we have was way too slow and I had to use several threads to
> speed things up and make it reasonable.
>

Before adding URL checking, it takes ~2mins.  I was able to keep the
runtime at 2mins 20sec by pulling in your code for threading and using
64 threads.

Matt

[Buildroot] [PATCH 01/14] support/scripts/pkg-stats: URL checking support

[Thomas Petazzoni]

Hello,

On Fri, 21 Sep 2018 07:56:46 -0500, Matthew Weber wrote:

> How does this look? (Still some bugs with some of the invalid sites,
> the webservers are reporting incorrect codes....)
> https://pste.eu/p/JpsH.html

Looks good!

> Before adding URL checking, it takes ~2mins.  I was able to keep the
> runtime at 2mins 20sec by pulling in your code for threading and using
> 64 threads.

I think my threading stuff had some issues, which were reported by
Ricardo, and are the reason why my "upstream version checking" stuff
was not merged.

I'd say, let's merge a first version without threading, and we'll see
at adding threading later.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 01/14] support/scripts/pkg-stats: URL checking support

[Matthew Weber]

Thomas,

On Fri, Sep 21, 2018 at 8:08 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Fri, 21 Sep 2018 07:56:46 -0500, Matthew Weber wrote:
>
> > How does this look? (Still some bugs with some of the invalid sites,
> > the webservers are reporting incorrect codes....)
> > https://pste.eu/p/JpsH.html
>
> Looks good!
>
> > Before adding URL checking, it takes ~2mins.  I was able to keep the
> > runtime at 2mins 20sec by pulling in your code for threading and using
> > 64 threads.
>
> I think my threading stuff had some issues, which were reported by
> Ricardo, and are the reason why my "upstream version checking" stuff
> was not merged.
>
> I'd say, let's merge a first version without threading, and we'll see
> at adding threading later.
>

Sounds good.  I'll submit it in two pieces, the first without
threading and the second, adding the threading as a general feature
you can add various tasks to and run them in parallel.

Matt

[Buildroot] [PATCH 12/14] boot/vexpress-firmware: URL update

[Peter Korsgaard]

>>>>> "Matt" == Matt Weber <matthew.weber@rockwellcollins.com> writes:

 > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>

Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 03/14] boot/at91bootstrap3: URL update

[Peter Korsgaard]

>>>>> "Matt" == Matt Weber <matthew.weber@rockwellcollins.com> writes:

 > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>

Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 06/14] package/android-tools: URL update

[Peter Korsgaard]

>>>>> "Matt" == Matt Weber <matthew.weber@rockwellcollins.com> writes:

 > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>

Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 07/14] package/arp-scan: URL update

[Peter Korsgaard]

>>>>> "Matt" == Matt Weber <matthew.weber@rockwellcollins.com> writes:

 > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>

Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 08/14] package/bandwidthd: URL move to newline

[Peter Korsgaard]

>>>>> "Matt" == Matt Weber <matthew.weber@rockwellcollins.com> writes:

 > Allows scripting of URL checking to be simplier
 > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>

Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 05/14] boot/xloader: URL update

[Peter Korsgaard]

>>>>> "Matt" == Matt Weber <matthew.weber@rockwellcollins.com> writes:

 > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>

Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 09/14] package/connman: URL move to newline

[Peter Korsgaard]

>>>>> "Matt" == Matt Weber <matthew.weber@rockwellcollins.com> writes:

 > Allows scripting of URL checking to be simplier
 > Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>

Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.

-- 
Bye, Peter Korsgaard