[Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess

[Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess

[Thomas Petazzoni]

The pkg-stats scripts tries to match packages against
release-monitoring.org in two days:

- First by using the "Buildroot" distribution registered on
  release-monitoring.org, in which we have added a lot of mappings
  between Buildroot package names and release-monitoring.org package
  names. If there is a match using this distribution, the package
  status is RM_API_STATUS_FOUND_BY_DISTRO, which means that the
  resulting HTML has a "found by distro" statement.

- Then, if the first solution didn't work, by using the pattern
  matching, as done in the check_package_get_latest_version_by_guess()
  function.

However, there is a bug in this later case: it sets the package status
to RM_API_STATUS_FOUND_BY_DISTRO as well, while it should have been
RM_API_STATUS_FOUND_BY_PATTERN. Due to this bug, in the resulting HTML
file from a pkg-stats run, all packages are marked as "found by
distro" even the ones that are "found by guess".

This commit fixes that by setting the correct package status.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
This should be backported to stable branches of Buildroot.
---
 support/scripts/pkg-stats | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index cc91d13167..0f1521873f 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -512,7 +512,7 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True):
 
             if len(projects) > 0:
                 check_package_latest_version_set_status(pkg,
-                                                        RM_API_STATUS_FOUND_BY_DISTRO,
+                                                        RM_API_STATUS_FOUND_BY_PATTERN,
                                                         projects[0]['version'],
                                                         projects[0]['id'])
                 return True
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org

[Thomas Petazzoni]

The pkg-stats script queries release-monitoring.org to find the latest
upstream versions of our packages. However, up until recently,
release-monitoring.org had no notion of stable
vs. development/release-candidate versions, so for some packages the
"latest" version was in fact a development/release-candidate version
that we didn't want to package in Buildroot.

However, in recent time, release-monitoring.org has gained support for
differentiating stable vs. development releases of upstream
projects. See for example
https://release-monitoring.org/project/10024/ for the glib library,
which has a number of versions marked "Pre-release".

The JSON blurb returned by release-monitoring.org has 3 relevant
fields:

 - "version", which we are using currently, which is a string
   containing the reference of the latest version, including
   pre-release.

 - "versions", which is an array of strings listing all versions,
   pre-release or not.

 - "stable_versions", which is an array of string listing only
   non-pre-release versions. It is ordered newest first to oldest
   last.

So, this commit changes from using 'version' to using
'stable_versions[0]'.

As an example, before this change, pkg-stats reports that nfs-utils
needs to be bumped to 2.5.5rc3, while after this patch, it reports
that nfs-utils is already at 2.5.4, and that this is the latest stable
version (modulo an issue where Buildroot has 2.5.4 and
release-monitoring.org has 2-5-4, this will be addressed separately).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
This should be backported to stable branches of Buildroot, so that the
pkg-stats results generated for those stable branches also benefit
from this logic.
---
 support/scripts/pkg-stats | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 0f1521873f..f7bc300770 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -484,7 +484,7 @@ async def check_package_get_latest_version_by_distro(session, pkg, retry=True):
                 return False
 
             data = await resp.json()
-            version = data['version'] if 'version' in data else None
+            version = data['stable_versions'][0] if 'stable_versions' in data else None
             check_package_latest_version_set_status(pkg,
                                                     RM_API_STATUS_FOUND_BY_DISTRO,
                                                     version,
@@ -507,13 +507,13 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True):
 
             data = await resp.json()
             # filter projects that have the right name and a version defined
-            projects = [p for p in data['projects'] if p['name'] == pkg.name and 'version' in p]
+            projects = [p for p in data['projects'] if p['name'] == pkg.name and 'stable_versions' in p]
             projects.sort(key=lambda x: x['id'])
 
             if len(projects) > 0:
                 check_package_latest_version_set_status(pkg,
                                                         RM_API_STATUS_FOUND_BY_PATTERN,
-                                                        projects[0]['version'],
+                                                        projects[0]['stable_versions'][0],
                                                         projects[0]['id'])
                 return True
 
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org

[Arnout Vandecappelle]

On 08/10/2021 23:21, Thomas Petazzoni wrote:
> The pkg-stats script queries release-monitoring.org to find the latest
> upstream versions of our packages. However, up until recently,
> release-monitoring.org had no notion of stable
> vs. development/release-candidate versions, so for some packages the
> "latest" version was in fact a development/release-candidate version
> that we didn't want to package in Buildroot.
> 
> However, in recent time, release-monitoring.org has gained support for
> differentiating stable vs. development releases of upstream
> projects. See for example
> https://release-monitoring.org/project/10024/ for the glib library,
> which has a number of versions marked "Pre-release".
> 
> The JSON blurb returned by release-monitoring.org has 3 relevant
> fields:
> 
>   - "version", which we are using currently, which is a string
>     containing the reference of the latest version, including
>     pre-release.
> 
>   - "versions", which is an array of strings listing all versions,
>     pre-release or not.
> 
>   - "stable_versions", which is an array of string listing only
>     non-pre-release versions. It is ordered newest first to oldest
>     last.

  So we can be sure that "stable_versions" is present? Yann committed an earlier 
version by François doing the same thing, but it falls back to 'version' if 
'stable_versions' doesn't exist.

> So, this commit changes from using 'version' to using
> 'stable_versions[0]'.
> 
> As an example, before this change, pkg-stats reports that nfs-utils
> needs to be bumped to 2.5.5rc3, while after this patch, it reports
> that nfs-utils is already at 2.5.4, and that this is the latest stable
> version (modulo an issue where Buildroot has 2.5.4 and
> release-monitoring.org has 2-5-4, this will be addressed separately).
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> This should be backported to stable branches of Buildroot, so that the
> pkg-stats results generated for those stable branches also benefit
> from this logic.
> ---
>   support/scripts/pkg-stats | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 0f1521873f..f7bc300770 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -484,7 +484,7 @@ async def check_package_get_latest_version_by_distro(session, pkg, retry=True):
>                   return False
>   
>               data = await resp.json()
> -            version = data['version'] if 'version' in data else None
> +            version = data['stable_versions'][0] if 'stable_versions' in data else None
>               check_package_latest_version_set_status(pkg,
>                                                       RM_API_STATUS_FOUND_BY_DISTRO,
>                                                       version,
> @@ -507,13 +507,13 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True):
>   
>               data = await resp.json()
>               # filter projects that have the right name and a version defined
> -            projects = [p for p in data['projects'] if p['name'] == pkg.name and 'version' in p]
> +            projects = [p for p in data['projects'] if p['name'] == pkg.name and 'stable_versions' in p]

  François' patch doesn't have this, so I applied this part to master, thanks.

  @Peter both François' commit and this one have to be applied to stable.

  Regards,
  Arnout

>               projects.sort(key=lambda x: x['id'])
>   
>               if len(projects) > 0:
>                   check_package_latest_version_set_status(pkg,
>                                                           RM_API_STATUS_FOUND_BY_PATTERN,
> -                                                        projects[0]['version'],
> +                                                        projects[0]['stable_versions'][0],
>                                                           projects[0]['id'])
>                   return True
>   
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org

[Thomas Petazzoni]

On Thu, 21 Oct 2021 21:14:18 +0200
Arnout Vandecappelle <arnout@mind.be> wrote:

> >   - "stable_versions", which is an array of string listing only
> >     non-pre-release versions. It is ordered newest first to oldest
> >     last.  
> 
>   So we can be sure that "stable_versions" is present? Yann committed an earlier 
> version by François doing the same thing, but it falls back to 'version' if 
> 'stable_versions' doesn't exist.

When I implemented this, I looked at the JSON blurb provided by
release-monitoring.org, and in all cases that I could see, the
"stable_versions" field was present.

I cannot be 100% sure though that it will *always* be present.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

 > The pkg-stats script queries release-monitoring.org to find the latest
 > upstream versions of our packages. However, up until recently,
 > release-monitoring.org had no notion of stable
 > vs. development/release-candidate versions, so for some packages the
 > "latest" version was in fact a development/release-candidate version
 > that we didn't want to package in Buildroot.

 > However, in recent time, release-monitoring.org has gained support for
 > differentiating stable vs. development releases of upstream
 > projects. See for example
 > https://release-monitoring.org/project/10024/ for the glib library,
 > which has a number of versions marked "Pre-release".

 > The JSON blurb returned by release-monitoring.org has 3 relevant
 > fields:

 >  - "version", which we are using currently, which is a string
 >    containing the reference of the latest version, including
 >    pre-release.

 >  - "versions", which is an array of strings listing all versions,
 >    pre-release or not.

 >  - "stable_versions", which is an array of string listing only
 >    non-pre-release versions. It is ordered newest first to oldest
 >    last.

 > So, this commit changes from using 'version' to using
 > 'stable_versions[0]'.

 > As an example, before this change, pkg-stats reports that nfs-utils
 > needs to be bumped to 2.5.5rc3, while after this patch, it reports
 > that nfs-utils is already at 2.5.4, and that this is the latest stable
 > version (modulo an issue where Buildroot has 2.5.4 and
 > release-monitoring.org has 2-5-4, this will be addressed separately).

 > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 > ---
 > This should be backported to stable branches of Buildroot, so that the
 > pkg-stats results generated for those stable branches also benefit
 > from this logic.

Committed to 2021.02.x and 2021.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess

[Arnout Vandecappelle]

On 08/10/2021 23:21, Thomas Petazzoni wrote:
> The pkg-stats scripts tries to match packages against
> release-monitoring.org in two days:
> 
> - First by using the "Buildroot" distribution registered on
>    release-monitoring.org, in which we have added a lot of mappings
>    between Buildroot package names and release-monitoring.org package
>    names. If there is a match using this distribution, the package
>    status is RM_API_STATUS_FOUND_BY_DISTRO, which means that the
>    resulting HTML has a "found by distro" statement.
> 
> - Then, if the first solution didn't work, by using the pattern
>    matching, as done in the check_package_get_latest_version_by_guess()
>    function.
> 
> However, there is a bug in this later case: it sets the package status
> to RM_API_STATUS_FOUND_BY_DISTRO as well, while it should have been
> RM_API_STATUS_FOUND_BY_PATTERN. Due to this bug, in the resulting HTML
> file from a pkg-stats run, all packages are marked as "found by
> distro" even the ones that are "found by guess".
> 
> This commit fixes that by setting the correct package status.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

  Applied to master, thanks.

  Regards,
  Arnout

> ---
> This should be backported to stable branches of Buildroot.
> ---
>   support/scripts/pkg-stats | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index cc91d13167..0f1521873f 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -512,7 +512,7 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True):
>   
>               if len(projects) > 0:
>                   check_package_latest_version_set_status(pkg,
> -                                                        RM_API_STATUS_FOUND_BY_DISTRO,
> +                                                        RM_API_STATUS_FOUND_BY_PATTERN,
>                                                           projects[0]['version'],
>                                                           projects[0]['id'])
>                   return True
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

 > The pkg-stats scripts tries to match packages against
 > release-monitoring.org in two days:

 > - First by using the "Buildroot" distribution registered on
 >   release-monitoring.org, in which we have added a lot of mappings
 >   between Buildroot package names and release-monitoring.org package
 >   names. If there is a match using this distribution, the package
 >   status is RM_API_STATUS_FOUND_BY_DISTRO, which means that the
 >   resulting HTML has a "found by distro" statement.

 > - Then, if the first solution didn't work, by using the pattern
 >   matching, as done in the check_package_get_latest_version_by_guess()
 >   function.

 > However, there is a bug in this later case: it sets the package status
 > to RM_API_STATUS_FOUND_BY_DISTRO as well, while it should have been
 > RM_API_STATUS_FOUND_BY_PATTERN. Due to this bug, in the resulting HTML
 > file from a pkg-stats run, all packages are marked as "found by
 > distro" even the ones that are "found by guess".

 > This commit fixes that by setting the correct package status.

 > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 > ---
 > This should be backported to stable branches of Buildroot.

Committed to 2021.02.x and 2021.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot