* [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess
@ 2021-10-08 21:21 Thomas Petazzoni
2021-10-08 21:21 ` [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org Thomas Petazzoni
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Thomas Petazzoni @ 2021-10-08 21:21 UTC (permalink / raw)
To: buildroot; +Cc: Thomas Petazzoni
The pkg-stats scripts tries to match packages against
release-monitoring.org in two days:
- First by using the "Buildroot" distribution registered on
release-monitoring.org, in which we have added a lot of mappings
between Buildroot package names and release-monitoring.org package
names. If there is a match using this distribution, the package
status is RM_API_STATUS_FOUND_BY_DISTRO, which means that the
resulting HTML has a "found by distro" statement.
- Then, if the first solution didn't work, by using the pattern
matching, as done in the check_package_get_latest_version_by_guess()
function.
However, there is a bug in this later case: it sets the package status
to RM_API_STATUS_FOUND_BY_DISTRO as well, while it should have been
RM_API_STATUS_FOUND_BY_PATTERN. Due to this bug, in the resulting HTML
file from a pkg-stats run, all packages are marked as "found by
distro" even the ones that are "found by guess".
This commit fixes that by setting the correct package status.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
This should be backported to stable branches of Buildroot.
---
support/scripts/pkg-stats | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index cc91d13167..0f1521873f 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -512,7 +512,7 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True):
if len(projects) > 0:
check_package_latest_version_set_status(pkg,
- RM_API_STATUS_FOUND_BY_DISTRO,
+ RM_API_STATUS_FOUND_BY_PATTERN,
projects[0]['version'],
projects[0]['id'])
return True
--
2.31.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org
2021-10-08 21:21 [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess Thomas Petazzoni
@ 2021-10-08 21:21 ` Thomas Petazzoni
2021-10-21 19:14 ` Arnout Vandecappelle
2021-10-25 18:54 ` Peter Korsgaard
2021-10-21 19:09 ` [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess Arnout Vandecappelle
2021-10-25 18:54 ` Peter Korsgaard
2 siblings, 2 replies; 7+ messages in thread
From: Thomas Petazzoni @ 2021-10-08 21:21 UTC (permalink / raw)
To: buildroot; +Cc: Thomas Petazzoni
The pkg-stats script queries release-monitoring.org to find the latest
upstream versions of our packages. However, up until recently,
release-monitoring.org had no notion of stable
vs. development/release-candidate versions, so for some packages the
"latest" version was in fact a development/release-candidate version
that we didn't want to package in Buildroot.
However, in recent time, release-monitoring.org has gained support for
differentiating stable vs. development releases of upstream
projects. See for example
https://release-monitoring.org/project/10024/ for the glib library,
which has a number of versions marked "Pre-release".
The JSON blurb returned by release-monitoring.org has 3 relevant
fields:
- "version", which we are using currently, which is a string
containing the reference of the latest version, including
pre-release.
- "versions", which is an array of strings listing all versions,
pre-release or not.
- "stable_versions", which is an array of string listing only
non-pre-release versions. It is ordered newest first to oldest
last.
So, this commit changes from using 'version' to using
'stable_versions[0]'.
As an example, before this change, pkg-stats reports that nfs-utils
needs to be bumped to 2.5.5rc3, while after this patch, it reports
that nfs-utils is already at 2.5.4, and that this is the latest stable
version (modulo an issue where Buildroot has 2.5.4 and
release-monitoring.org has 2-5-4, this will be addressed separately).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
This should be backported to stable branches of Buildroot, so that the
pkg-stats results generated for those stable branches also benefit
from this logic.
---
support/scripts/pkg-stats | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 0f1521873f..f7bc300770 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -484,7 +484,7 @@ async def check_package_get_latest_version_by_distro(session, pkg, retry=True):
return False
data = await resp.json()
- version = data['version'] if 'version' in data else None
+ version = data['stable_versions'][0] if 'stable_versions' in data else None
check_package_latest_version_set_status(pkg,
RM_API_STATUS_FOUND_BY_DISTRO,
version,
@@ -507,13 +507,13 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True):
data = await resp.json()
# filter projects that have the right name and a version defined
- projects = [p for p in data['projects'] if p['name'] == pkg.name and 'version' in p]
+ projects = [p for p in data['projects'] if p['name'] == pkg.name and 'stable_versions' in p]
projects.sort(key=lambda x: x['id'])
if len(projects) > 0:
check_package_latest_version_set_status(pkg,
RM_API_STATUS_FOUND_BY_PATTERN,
- projects[0]['version'],
+ projects[0]['stable_versions'][0],
projects[0]['id'])
return True
--
2.31.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess
2021-10-08 21:21 [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess Thomas Petazzoni
2021-10-08 21:21 ` [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org Thomas Petazzoni
@ 2021-10-21 19:09 ` Arnout Vandecappelle
2021-10-25 18:54 ` Peter Korsgaard
2 siblings, 0 replies; 7+ messages in thread
From: Arnout Vandecappelle @ 2021-10-21 19:09 UTC (permalink / raw)
To: Thomas Petazzoni, buildroot, Peter Korsgaard
On 08/10/2021 23:21, Thomas Petazzoni wrote:
> The pkg-stats scripts tries to match packages against
> release-monitoring.org in two days:
>
> - First by using the "Buildroot" distribution registered on
> release-monitoring.org, in which we have added a lot of mappings
> between Buildroot package names and release-monitoring.org package
> names. If there is a match using this distribution, the package
> status is RM_API_STATUS_FOUND_BY_DISTRO, which means that the
> resulting HTML has a "found by distro" statement.
>
> - Then, if the first solution didn't work, by using the pattern
> matching, as done in the check_package_get_latest_version_by_guess()
> function.
>
> However, there is a bug in this later case: it sets the package status
> to RM_API_STATUS_FOUND_BY_DISTRO as well, while it should have been
> RM_API_STATUS_FOUND_BY_PATTERN. Due to this bug, in the resulting HTML
> file from a pkg-stats run, all packages are marked as "found by
> distro" even the ones that are "found by guess".
>
> This commit fixes that by setting the correct package status.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Applied to master, thanks.
Regards,
Arnout
> ---
> This should be backported to stable branches of Buildroot.
> ---
> support/scripts/pkg-stats | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index cc91d13167..0f1521873f 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -512,7 +512,7 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True):
>
> if len(projects) > 0:
> check_package_latest_version_set_status(pkg,
> - RM_API_STATUS_FOUND_BY_DISTRO,
> + RM_API_STATUS_FOUND_BY_PATTERN,
> projects[0]['version'],
> projects[0]['id'])
> return True
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org
2021-10-08 21:21 ` [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org Thomas Petazzoni
@ 2021-10-21 19:14 ` Arnout Vandecappelle
2021-10-21 20:03 ` Thomas Petazzoni
2021-10-25 18:54 ` Peter Korsgaard
1 sibling, 1 reply; 7+ messages in thread
From: Arnout Vandecappelle @ 2021-10-21 19:14 UTC (permalink / raw)
To: Thomas Petazzoni, buildroot; +Cc: Francois Perrad, Yann E. MORIN
On 08/10/2021 23:21, Thomas Petazzoni wrote:
> The pkg-stats script queries release-monitoring.org to find the latest
> upstream versions of our packages. However, up until recently,
> release-monitoring.org had no notion of stable
> vs. development/release-candidate versions, so for some packages the
> "latest" version was in fact a development/release-candidate version
> that we didn't want to package in Buildroot.
>
> However, in recent time, release-monitoring.org has gained support for
> differentiating stable vs. development releases of upstream
> projects. See for example
> https://release-monitoring.org/project/10024/ for the glib library,
> which has a number of versions marked "Pre-release".
>
> The JSON blurb returned by release-monitoring.org has 3 relevant
> fields:
>
> - "version", which we are using currently, which is a string
> containing the reference of the latest version, including
> pre-release.
>
> - "versions", which is an array of strings listing all versions,
> pre-release or not.
>
> - "stable_versions", which is an array of string listing only
> non-pre-release versions. It is ordered newest first to oldest
> last.
So we can be sure that "stable_versions" is present? Yann committed an earlier
version by François doing the same thing, but it falls back to 'version' if
'stable_versions' doesn't exist.
> So, this commit changes from using 'version' to using
> 'stable_versions[0]'.
>
> As an example, before this change, pkg-stats reports that nfs-utils
> needs to be bumped to 2.5.5rc3, while after this patch, it reports
> that nfs-utils is already at 2.5.4, and that this is the latest stable
> version (modulo an issue where Buildroot has 2.5.4 and
> release-monitoring.org has 2-5-4, this will be addressed separately).
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> This should be backported to stable branches of Buildroot, so that the
> pkg-stats results generated for those stable branches also benefit
> from this logic.
> ---
> support/scripts/pkg-stats | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 0f1521873f..f7bc300770 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -484,7 +484,7 @@ async def check_package_get_latest_version_by_distro(session, pkg, retry=True):
> return False
>
> data = await resp.json()
> - version = data['version'] if 'version' in data else None
> + version = data['stable_versions'][0] if 'stable_versions' in data else None
> check_package_latest_version_set_status(pkg,
> RM_API_STATUS_FOUND_BY_DISTRO,
> version,
> @@ -507,13 +507,13 @@ async def check_package_get_latest_version_by_guess(session, pkg, retry=True):
>
> data = await resp.json()
> # filter projects that have the right name and a version defined
> - projects = [p for p in data['projects'] if p['name'] == pkg.name and 'version' in p]
> + projects = [p for p in data['projects'] if p['name'] == pkg.name and 'stable_versions' in p]
François' patch doesn't have this, so I applied this part to master, thanks.
@Peter both François' commit and this one have to be applied to stable.
Regards,
Arnout
> projects.sort(key=lambda x: x['id'])
>
> if len(projects) > 0:
> check_package_latest_version_set_status(pkg,
> RM_API_STATUS_FOUND_BY_PATTERN,
> - projects[0]['version'],
> + projects[0]['stable_versions'][0],
> projects[0]['id'])
> return True
>
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org
2021-10-21 19:14 ` Arnout Vandecappelle
@ 2021-10-21 20:03 ` Thomas Petazzoni
0 siblings, 0 replies; 7+ messages in thread
From: Thomas Petazzoni @ 2021-10-21 20:03 UTC (permalink / raw)
To: Arnout Vandecappelle; +Cc: Francois Perrad, Yann E. MORIN, buildroot
On Thu, 21 Oct 2021 21:14:18 +0200
Arnout Vandecappelle <arnout@mind.be> wrote:
> > - "stable_versions", which is an array of string listing only
> > non-pre-release versions. It is ordered newest first to oldest
> > last.
>
> So we can be sure that "stable_versions" is present? Yann committed an earlier
> version by François doing the same thing, but it falls back to 'version' if
> 'stable_versions' doesn't exist.
When I implemented this, I looked at the JSON blurb provided by
release-monitoring.org, and in all cases that I could see, the
"stable_versions" field was present.
I cannot be 100% sure though that it will *always* be present.
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess
2021-10-08 21:21 [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess Thomas Petazzoni
2021-10-08 21:21 ` [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org Thomas Petazzoni
2021-10-21 19:09 ` [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess Arnout Vandecappelle
@ 2021-10-25 18:54 ` Peter Korsgaard
2 siblings, 0 replies; 7+ messages in thread
From: Peter Korsgaard @ 2021-10-25 18:54 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: buildroot
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:
> The pkg-stats scripts tries to match packages against
> release-monitoring.org in two days:
> - First by using the "Buildroot" distribution registered on
> release-monitoring.org, in which we have added a lot of mappings
> between Buildroot package names and release-monitoring.org package
> names. If there is a match using this distribution, the package
> status is RM_API_STATUS_FOUND_BY_DISTRO, which means that the
> resulting HTML has a "found by distro" statement.
> - Then, if the first solution didn't work, by using the pattern
> matching, as done in the check_package_get_latest_version_by_guess()
> function.
> However, there is a bug in this later case: it sets the package status
> to RM_API_STATUS_FOUND_BY_DISTRO as well, while it should have been
> RM_API_STATUS_FOUND_BY_PATTERN. Due to this bug, in the resulting HTML
> file from a pkg-stats run, all packages are marked as "found by
> distro" even the ones that are "found by guess".
> This commit fixes that by setting the correct package status.
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> This should be backported to stable branches of Buildroot.
Committed to 2021.02.x and 2021.08.x, thanks.
--
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org
2021-10-08 21:21 ` [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org Thomas Petazzoni
2021-10-21 19:14 ` Arnout Vandecappelle
@ 2021-10-25 18:54 ` Peter Korsgaard
1 sibling, 0 replies; 7+ messages in thread
From: Peter Korsgaard @ 2021-10-25 18:54 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: buildroot
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:
> The pkg-stats script queries release-monitoring.org to find the latest
> upstream versions of our packages. However, up until recently,
> release-monitoring.org had no notion of stable
> vs. development/release-candidate versions, so for some packages the
> "latest" version was in fact a development/release-candidate version
> that we didn't want to package in Buildroot.
> However, in recent time, release-monitoring.org has gained support for
> differentiating stable vs. development releases of upstream
> projects. See for example
> https://release-monitoring.org/project/10024/ for the glib library,
> which has a number of versions marked "Pre-release".
> The JSON blurb returned by release-monitoring.org has 3 relevant
> fields:
> - "version", which we are using currently, which is a string
> containing the reference of the latest version, including
> pre-release.
> - "versions", which is an array of strings listing all versions,
> pre-release or not.
> - "stable_versions", which is an array of string listing only
> non-pre-release versions. It is ordered newest first to oldest
> last.
> So, this commit changes from using 'version' to using
> 'stable_versions[0]'.
> As an example, before this change, pkg-stats reports that nfs-utils
> needs to be bumped to 2.5.5rc3, while after this patch, it reports
> that nfs-utils is already at 2.5.4, and that this is the latest stable
> version (modulo an issue where Buildroot has 2.5.4 and
> release-monitoring.org has 2-5-4, this will be addressed separately).
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> This should be backported to stable branches of Buildroot, so that the
> pkg-stats results generated for those stable branches also benefit
> from this logic.
Committed to 2021.02.x and 2021.08.x, thanks.
--
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-10-25 18:55 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-08 21:21 [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess Thomas Petazzoni
2021-10-08 21:21 ` [Buildroot] [PATCH 2/2] support/scripts/pkg-stats: use the new 'stable_versions' field of release-monitoring.org Thomas Petazzoni
2021-10-21 19:14 ` Arnout Vandecappelle
2021-10-21 20:03 ` Thomas Petazzoni
2021-10-25 18:54 ` Peter Korsgaard
2021-10-21 19:09 ` [Buildroot] [PATCH 1/2] support/scripts/pkg-stats: fix the status for packages found by guess Arnout Vandecappelle
2021-10-25 18:54 ` Peter Korsgaard
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.