[Buildroot] [PATCH v3 0/3] Use aiohttp in pkg-stats

[Thomas Petazzoni <thomas.petazzoni@bootlin.com>]

Hello,

I started investigating why pkg-stats was sometimes stuck on the
server running it on a daily basis to populate
autobuild.buildroot.org/stats/ and send the autobuilder e-mails. The
subprocesses started by "multiprocessing" to retrieve the latest
upstream version from release-monitoring.org were stuck holding a
lock. Without providing a definitive conclusion, some preliminary
research showed that multiprocessing can be tricky and cause some
issues with locks.

Discussing this with Titouan, he suggested to use aiohttp instead of
multiprocessing. And indeed, it makes a lot of sense to use this
popular asynchronous HTTP library.

This patch series switches the latest version retrieval and the
upstream URL checking to aiohttp, and as a bonus adds some logging to
show the progress of the retrieval, as it can be quite long.

Changes since v2:

 - Use python3 in the shebang

 - Use asyncio.TimeoutError in the exception handling

 - Slightly rework how packages with "no valid infra" are handled in
   the "latest version" check, but we keep a loop to handle such
   packages before the main loop, as we want the real count of
   packages that the main loop will handle.

 - Use asyncio.get_event_loop() + loop.run_until_complete() instead of
   asyncio.run().

Changes since v1:

 - Pass trust_env=True when creating the aiohttp.ClientSession() so
   that HTTP proxy environment variables are taken into
   account. Suggested by Matt Weber.

 - Fix bogus indentation that was breaking the logic of the latest
   version retrieval.

Thanks,

Thomas


Thomas Petazzoni (3):
  support/scripts/pkg-stats: use aiohttp for latest version retrieval
  support/scripts/pkg-stats: use aiohttp for upstream URL checking
  support/scripts/pkg-stats: show progress of upstream URL and latest
    version

 support/scripts/pkg-stats | 213 ++++++++++++++++++++++----------------
 1 file changed, 126 insertions(+), 87 deletions(-)

-- 
2.26.2