[Buildroot] [PATCH 1/2] pkg-stats: query release-monitoring.org with upstream name

[Buildroot] [PATCH 1/2] pkg-stats: query release-monitoring.org with upstream name

[Francois Perrad]

By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables *_DISTNAME
give the CPAN name which is known by release-monitoring.org

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
---
 support/scripts/pkg-stats | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 992c2dd7c..e03f714a6 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -47,6 +47,7 @@ class Package:
     all_licenses = list()
     all_license_files = list()
     all_versions = dict()
+    all_upstream_names = dict()
 
     def __init__(self, name, path):
         self.name = name
@@ -58,6 +59,7 @@ class Package:
         self.patch_count = 0
         self.warnings = 0
         self.current_version = None
+        self.upstream_name = name
         self.url = None
         self.url_status = None
         self.url_worker = None
@@ -134,6 +136,14 @@ class Package:
         if var in self.all_versions:
             self.current_version = self.all_versions[var]
 
+    def set_upstream_name(self):
+        """
+        Fills in the .upstream_name field
+        """
+        var = self.pkgvar()
+        if var in self.all_upstream_names:
+            self.upstream_name = self.all_upstream_names[var]
+
     def set_check_package_warnings(self):
         """
         Fills in the .warnings field
@@ -227,7 +237,7 @@ def get_pkglist(npackages, package_list):
 def package_init_make_info():
     # Fetch all variables at once
     variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars",
-                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION"])
+                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME"])
     variable_list = variables.splitlines()
 
     # We process first the host package VERSION, and then the target
@@ -261,6 +271,11 @@ def package_init_make_info():
             pkgvar = pkgvar[:-8]
             Package.all_versions[pkgvar] = value
 
+        elif pkgvar.endswith("_DISTNAME"):
+            # exist only with the perl-package infra
+            pkgvar = pkgvar[:-9]
+            Package.all_upstream_names[pkgvar] = value
+
 
 def check_url_status_worker(url, url_status):
     if url_status != "Missing" and url_status != "No Config.in":
@@ -349,7 +364,7 @@ def check_package_latest_version(packages):
                                     cert_reqs='CERT_REQUIRED', ca_certs=certifi.where(),
                                     timeout=30)
     worker_pool = Pool(processes=64)
-    results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages))
+    results = worker_pool.map(check_package_latest_version_worker, (pkg.upstream_name for pkg in packages))
     for pkg, r in zip(packages, results):
         pkg.latest_version = r
     del http_pool
@@ -742,6 +757,7 @@ def __main__():
         pkg.set_check_package_warnings()
         pkg.set_current_version()
         pkg.set_url()
+        pkg.set_upstream_name()
     print("Checking URL status")
     check_package_urls(packages)
     print("Getting latest versions ...")
-- 
2.20.1

[Buildroot] [PATCH 2/2] pkg-stats: more upstream names

[Francois Perrad]

the python infrastructure populates the variables _NAME_UPSTREAM
(like the luarocks one).
and pkg-stats uses them.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
---
 package/pkg-python.mk     | 1 +
 support/scripts/pkg-stats | 7 ++++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index e90692041..7ec6f215a 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -108,6 +108,7 @@ define inner-python-package
 $(2)_ENV         ?=
 $(2)_BUILD_OPTS   ?=
 $(2)_INSTALL_OPTS ?=
+$(2)_NAME_UPSTREAM ?= $(subst python-,,$(1))
 
 ifndef $(2)_SETUP_TYPE
  ifdef $(3)_SETUP_TYPE
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index e03f714a6..92159c30f 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -237,7 +237,7 @@ def get_pkglist(npackages, package_list):
 def package_init_make_info():
     # Fetch all variables at once
     variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars",
-                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME"])
+                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME %_NAME_UPSTREAM"])
     variable_list = variables.splitlines()
 
     # We process first the host package VERSION, and then the target
@@ -276,6 +276,11 @@ def package_init_make_info():
             pkgvar = pkgvar[:-9]
             Package.all_upstream_names[pkgvar] = value
 
+        elif pkgvar.endswith("_NAME_UPSTREAM"):
+            # populated by some infra: luarocks, python
+            pkgvar = pkgvar[:-14]
+            Package.all_upstream_names[pkgvar] = value
+
 
 def check_url_status_worker(url, url_status):
     if url_status != "Missing" and url_status != "No Config.in":
-- 
2.20.1

[Buildroot] [PATCH 1/2] pkg-stats: query release-monitoring.org with upstream name

[Thomas Petazzoni]

Hello,

+Victor Huesca in Cc.

On Wed, 28 Aug 2019 09:02:50 +0200
Francois Perrad <fperrad@gmail.com> wrote:

> By default, this upstream name is our BR name.
> But with perl-package infrastructure, the variables *_DISTNAME
> give the CPAN name which is known by release-monitoring.org
> 
> Signed-off-by: Francois Perrad <francois.perrad@gadz.org>

Thanks for implementing this. It matches what we discussed. The only
possible issue that I can see is if a package defines FOO_DISTNAME for
some internal reason, but is not a perl-package. Then we will pick up
whatever DISTNAME contains.

Perhaps we should standardize on FOO_NAME_UPSTREAM for all package
infrastructures including perl-package, and make it a "reserved"
variable name, so that it is forbidden to use it for anything else ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/2] pkg-stats: more upstream names

[Thomas Petazzoni]

Hello Fran?ois,

+Victor Huesca in Cc.

On Wed, 28 Aug 2019 09:02:51 +0200
Francois Perrad <fperrad@gmail.com> wrote:

> the python infrastructure populates the variables _NAME_UPSTREAM
> (like the luarocks one).
> and pkg-stats uses them.
> 
> Signed-off-by: Francois Perrad <francois.perrad@gadz.org>

I think this should be split into two parts:

 - one changing pkg-stats to use NAME_UPSTREAM, as it already helps for
   luarocks-package. However, could you name a few luarocks packages
   where their NAME_UPSTREAM matches the name of the project on
   release-monitoring.org ?

 - for the Python case, what should "upstream name" match ? The name
   registered on PyPi ? Sometimes the name registered on PyPi doesn't
   always exactly match the upstream name.

   In any case, the new NAME_UPSTREAM variable in python-package should
   be documented in the manual.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] pkg-stats: query release-monitoring.org with upstream name

[François Perrad]

Le mer. 28 ao?t 2019 ? 14:44, Thomas Petazzoni <thomas.petazzoni@bootlin.com>
a ?crit :

> Hello,
>
> +Victor Huesca in Cc.
>
> On Wed, 28 Aug 2019 09:02:50 +0200
> Francois Perrad <fperrad@gmail.com> wrote:
>
> > By default, this upstream name is our BR name.
> > But with perl-package infrastructure, the variables *_DISTNAME
> > give the CPAN name which is known by release-monitoring.org
> >
> > Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
>
> Thanks for implementing this. It matches what we discussed. The only
> possible issue that I can see is if a package defines FOO_DISTNAME for
> some internal reason, but is not a perl-package. Then we will pick up
> whatever DISTNAME contains.
>
> Perhaps we should standardize on FOO_NAME_UPSTREAM for all package
> infrastructures including perl-package, and make it a "reserved"
> variable name, so that it is forbidden to use it for anything else ?
>
>
Sounds great.
If we have our own FOO_NAME_UPSTREAM, that avoids to register "mapping" on
release-monitoring.org for any package.

When this addition will be validated, I'll cook one patch which updates
(s/DISTNAME/UPSTREAM_NAME/) the 110 Perl/CPAN packages and a V2 of this
patch.

Fran?ois

Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190901/73976526/attachment.html>

[Buildroot] [PATCH 2/2] pkg-stats: more upstream names

[François Perrad]

Le mer. 28 ao?t 2019 ? 14:47, Thomas Petazzoni <thomas.petazzoni@bootlin.com>
a ?crit :

> Hello Fran?ois,
>
> +Victor Huesca in Cc.
>
> On Wed, 28 Aug 2019 09:02:51 +0200
> Francois Perrad <fperrad@gmail.com> wrote:
>
> > the python infrastructure populates the variables _NAME_UPSTREAM
> > (like the luarocks one).
> > and pkg-stats uses them.
> >
> > Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
>
> I think this should be split into two parts:
>
>  - one changing pkg-stats to use NAME_UPSTREAM, as it already helps for
>    luarocks-package. However, could you name a few luarocks packages
>    where their NAME_UPSTREAM matches the name of the project on
>    release-monitoring.org ?
>

lua-basexx (basexx) see https://release-monitoring.org/project/15672/
lua-lpeg-patterns (lpeg_patterns) see
https://release-monitoring.org/project/15670/
lua-stdlib (stdlib) see https://release-monitoring.org/project/11735/

There is another issue with LuaRocks packages:
There is already a mismatch between the rock version and the upstream
version found by release-monitoring, for example:
    lua-http 0.3-0 vs 0.3
    lua-iconv 7-1 (https://luarocks.org/modules/luarocks/lua-iconv/7-1) vs
7 (https://release-monitoring.org/project/11668/)
This issue could be solve only by adding a specific LuaRocks backend to
release-monitoring.

Fran?ois


>  - for the Python case, what should "upstream name" match ? The name
>    registered on PyPi ? Sometimes the name registered on PyPi doesn't
>    always exactly match the upstream name.
>
>    In any case, the new NAME_UPSTREAM variable in python-package should
>    be documented in the manual.
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190901/80dd9a5f/attachment.html>

[Buildroot] [PATCH 2/2] pkg-stats: more upstream names

[Thomas Petazzoni]

On Wed, 28 Aug 2019 09:02:51 +0200
Francois Perrad <fperrad@gmail.com> wrote:

> the python infrastructure populates the variables _NAME_UPSTREAM
> (like the luarocks one).
> and pkg-stats uses them.
> 
> Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
> ---
>  package/pkg-python.mk     | 1 +
>  support/scripts/pkg-stats | 7 ++++++-
>  2 files changed, 7 insertions(+), 1 deletion(-)

So we discussed this with Arnout, and since release-monitoring.org
provides this mapping functionality, and someone has already filled in
the mapping details for almost all Buildroot packages. We don't want to
duplicate that information in Buildroot itself.

However, we can improve a bit the logic in pkg-stats when querying by
"guess". I'll reply to your PATCH 1/2 with more details about this idea.

In the mean time, I've marked this particular patch as Rejected.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] pkg-stats: query release-monitoring.org with upstream name

[Thomas Petazzoni]

Hello,

On Wed, 28 Aug 2019 09:02:50 +0200
Francois Perrad <fperrad@gmail.com> wrote:

> By default, this upstream name is our BR name.
> But with perl-package infrastructure, the variables *_DISTNAME
> give the CPAN name which is known by release-monitoring.org
> 
> Signed-off-by: Francois Perrad <francois.perrad@gadz.org>

Since we have this information already in Buildroot, it makes sense to
use it. However, the search on release-monitoring.org is done in two
steps:

 - Using the Buildroot distribution in release-monitoring and its
   mapping concept. In this case, we really want to use the Buildroot
   package name unchanged (which is not what your patch does)

 - Then, if no result is found through the Buildroot distribution
   above, we search in the entire namespace. Here, we should search
   with _DISTNAME when available. We could also improve the logic to
   strip python- and lua- in the front of packages, to help the guess
   to work.

Would you be willing to work on this ? It's not too far from the patch
you have already implemented here.

Thanks a lot!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] pkg-stats: query release-monitoring.org with upstream name

[François Perrad]

Le mer. 5 f?vr. 2020 ? 18:17, Thomas Petazzoni <thomas.petazzoni@bootlin.com>
a ?crit :

> Hello,
>
> On Wed, 28 Aug 2019 09:02:50 +0200
> Francois Perrad <fperrad@gmail.com> wrote:
>
> > By default, this upstream name is our BR name.
> > But with perl-package infrastructure, the variables *_DISTNAME
> > give the CPAN name which is known by release-monitoring.org
> >
> > Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
>
> Since we have this information already in Buildroot, it makes sense to
> use it. However, the search on release-monitoring.org is done in two
> steps:
>
>  - Using the Buildroot distribution in release-monitoring and its
>    mapping concept. In this case, we really want to use the Buildroot
>    package name unchanged (which is not what your patch does)
>
>
It means that a part of Buildroot source (or configuration) is hosted on
release-monitoring.org instead of our own git repository.


>  - Then, if no result is found through the Buildroot distribution
>    above, we search in the entire namespace. Here, we should search
>    with _DISTNAME when available. We could also improve the logic to
>    strip python- and lua- in the front of packages, to help the guess
>    to work.
>
> Would you be willing to work on this ? It's not too far from the patch
> you have already implemented here.
>
>
Another point about release-monitoring.org,
how to exclude `dev` branches when the policy of the upstream project is to
name them with an odd minor number ?

Here, some false positives:
                          perl |  DISTRO  |
https://release-monitoring.org/project/13599 | 5.30.1       | 5.31.8
|
                     webkitgtk |  DISTRO  |
https://release-monitoring.org/project/05355 | 2.26.3       | 2.27.4
|
                     wpewebkit |  DISTRO  |
https://release-monitoring.org/project/17557 | 2.26.3       | 2.27.4
|

Fran?ois

Thanks a lot!
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200207/ad86134b/attachment.html>

[Buildroot] [PATCH 1/2] pkg-stats: query release-monitoring.org with upstream name

[Thomas Petazzoni]

Hello Fran?ois,

On Fri, 7 Feb 2020 17:58:57 +0100
Fran?ois Perrad <francois.perrad@gadz.org> wrote:

> >  - Using the Buildroot distribution in release-monitoring and its
> >    mapping concept. In this case, we really want to use the Buildroot
> >    package name unchanged (which is not what your patch does)
>
> It means that a part of Buildroot source (or configuration) is hosted on
> release-monitoring.org instead of our own git repository.

Not really. What is stored on release-monitoring side is the mapping of
Buildroot package name to release-monitoring.org project names.

If we were to store this information on the Buildroot side, we could
say "it's a release-monitoring.org information that is stored inside
the Buildroot source code".

> Another point about release-monitoring.org,
> how to exclude `dev` branches when the policy of the upstream project is to
> name them with an odd minor number ?

This is a known issue, see
https://github.com/fedora-infra/anitya/issues/753. It would be great if
someone could work on this.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com