[PATCH 1/2] recipetool: Handle unclean response in go resolver

[PATCH 1/2] recipetool: Handle unclean response in go resolver

[Sven Schwermer]

From: Sven Schwermer <sven.schwermer@disruptive-technologies.com>

It appears that some go modules repond with a 404 error when trying to
resolve them dynamically. The response body may still contain the
go-import meta tag. An example for such behaviour is gonum.org/v1/gonum.

Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
---
 scripts/lib/recipetool/create_go.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py
index c560831442..0fb7115e26 100644
--- a/scripts/lib/recipetool/create_go.py
+++ b/scripts/lib/recipetool/create_go.py
@@ -16,7 +16,7 @@ from html.parser import HTMLParser
 from recipetool.create import RecipeHandler, handle_license_vars
 from recipetool.create import guess_license, tidy_licenses, fixup_license
 from recipetool.create import determine_from_url
-from urllib.error import URLError
+from urllib.error import URLError, HTTPError
 
 import bb.utils
 import json
@@ -251,15 +251,18 @@ class GoRecipeHandler(RecipeHandler):
         req = urllib.request.Request(url)
 
         try:
-            resp = urllib.request.urlopen(req)
-
+            body = urllib.request.urlopen(req).read()
+        except HTTPError as http_err:
+            logger.warning(
+                "Unclean status when fetching page from [%s]: %s", url, str(http_err))
+            body = http_err.fp.read()
         except URLError as url_err:
             logger.warning(
                 "Failed to fetch page from [%s]: %s", url, str(url_err))
             return None
 
         parser = GoImportHTMLParser()
-        parser.feed(resp.read().decode('utf-8'))
+        parser.feed(body.decode('utf-8'))
         parser.close()
 
         return GoImport(parser.import_prefix, parser.vcs, parser.repourl, None)
-- 
2.44.0

[PATCH 2/2] recipetool: Handle several go-import tags in go resolver

[Sven Schwermer]

From: Sven Schwermer <sven.schwermer@disruptive-technologies.com>

When dynamically resolving go modules, the HTML page may contain several
go-import meta tags. We must handle all and pick the correct one based
on the module name. An example for such a behaviour is
gonum.org/v1/gonum:

<meta name="go-import" content="gonum.org/v1/exp git https://github.com/gonum/exp">
<meta name="go-import" content="gonum.org/v1/gonum git https://github.com/gonum/gonum">
<meta name="go-import" content="gonum.org/v1/hdf5 git https://github.com/gonum/hdf5">
<meta name="go-import" content="gonum.org/v1/netlib git https://github.com/gonum/netlib">
<meta name="go-import" content="gonum.org/v1/plot git https://github.com/gonum/plot">
<meta name="go-import" content="gonum.org/v1/tools git https://github.com/gonum/tools">

Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
---
 scripts/lib/recipetool/create_go.py | 23 +++++++++--------------
 1 file changed, 9 insertions(+), 14 deletions(-)

diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py
index 0fb7115e26..a85a2f2786 100644
--- a/scripts/lib/recipetool/create_go.py
+++ b/scripts/lib/recipetool/create_go.py
@@ -225,7 +225,7 @@ class GoRecipeHandler(RecipeHandler):
 
             def __init__(self):
                 super().__init__()
-                self.__srv = []
+                self.__srv = {}
 
             def handle_starttag(self, tag, attrs):
                 if tag == 'meta' and list(
@@ -233,19 +233,14 @@ class GoRecipeHandler(RecipeHandler):
                     content = list(
                         filter(lambda a: (a[0] == 'content'), attrs))
                     if content:
-                        self.__srv = content[0][1].split()
+                        srv = content[0][1].split()
+                        self.__srv[srv[0]] = srv
 
-            @property
-            def import_prefix(self):
-                return self.__srv[0] if len(self.__srv) else None
-
-            @property
-            def vcs(self):
-                return self.__srv[1] if len(self.__srv) else None
-
-            @property
-            def repourl(self):
-                return self.__srv[2] if len(self.__srv) else None
+            def go_import(self, modulepath):
+                if modulepath in self.__srv:
+                    srv = self.__srv[modulepath]
+                    return GoImport(srv[0], srv[1], srv[2], None)
+                return None
 
         url = url.geturl() + "?go-get=1"
         req = urllib.request.Request(url)
@@ -265,7 +260,7 @@ class GoRecipeHandler(RecipeHandler):
         parser.feed(body.decode('utf-8'))
         parser.close()
 
-        return GoImport(parser.import_prefix, parser.vcs, parser.repourl, None)
+        return parser.go_import(modulepath)
 
     def __resolve_from_golang_proxy(self, modulepath, version):
         """
-- 
2.44.0

Re: [OE-core] [PATCH 2/2] recipetool: Handle several go-import tags in go resolver

[Richard Purdie]

On Thu, 2024-04-11 at 12:10 +0200, Sven via lists.openembedded.org wrote:
> From: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
> 
> When dynamically resolving go modules, the HTML page may contain several
> go-import meta tags. We must handle all and pick the correct one based
> on the module name. An example for such a behaviour is
> gonum.org/v1/gonum:
> 
> <meta name="go-import" content="gonum.org/v1/exp git https://github.com/gonum/exp">
> <meta name="go-import" content="gonum.org/v1/gonum git https://github.com/gonum/gonum">
> <meta name="go-import" content="gonum.org/v1/hdf5 git https://github.com/gonum/hdf5">
> <meta name="go-import" content="gonum.org/v1/netlib git https://github.com/gonum/netlib">
> <meta name="go-import" content="gonum.org/v1/plot git https://github.com/gonum/plot">
> <meta name="go-import" content="gonum.org/v1/tools git https://github.com/gonum/tools">
> 
> Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
> ---
>  scripts/lib/recipetool/create_go.py | 23 +++++++++--------------
>  1 file changed, 9 insertions(+), 14 deletions(-)
> 
> diff --git a/scripts/lib/recipetool/create_go.py b/scripts/lib/recipetool/create_go.py
> index 0fb7115e26..a85a2f2786 100644
> --- a/scripts/lib/recipetool/create_go.py
> +++ b/scripts/lib/recipetool/create_go.py
> @@ -225,7 +225,7 @@ class GoRecipeHandler(RecipeHandler):
>  
>              def __init__(self):
>                  super().__init__()
> -                self.__srv = []
> +                self.__srv = {}
>  
>              def handle_starttag(self, tag, attrs):
>                  if tag == 'meta' and list(
> @@ -233,19 +233,14 @@ class GoRecipeHandler(RecipeHandler):
>                      content = list(
>                          filter(lambda a: (a[0] == 'content'), attrs))
>                      if content:
> -                        self.__srv = content[0][1].split()
> +                        srv = content[0][1].split()
> +                        self.__srv[srv[0]] = srv
>  
> -            @property
> -            def import_prefix(self):
> -                return self.__srv[0] if len(self.__srv) else None
> -
> -            @property
> -            def vcs(self):
> -                return self.__srv[1] if len(self.__srv) else None
> -
> -            @property
> -            def repourl(self):
> -                return self.__srv[2] if len(self.__srv) else None
> +            def go_import(self, modulepath):
> +                if modulepath in self.__srv:
> +                    srv = self.__srv[modulepath]
> +                    return GoImport(srv[0], srv[1], srv[2], None)
> +                return None
>  
>          url = url.geturl() + "?go-get=1"
>          req = urllib.request.Request(url)
> @@ -265,7 +260,7 @@ class GoRecipeHandler(RecipeHandler):
>          parser.feed(body.decode('utf-8'))
>          parser.close()
>  
> -        return GoImport(parser.import_prefix, parser.vcs, parser.repourl, None)
> +        return parser.go_import(modulepath)
>  
>      def __resolve_from_golang_proxy(self, modulepath, version):
>          """

Should we be extending "oe-selftest -r recipetool" with tests for something?

Cheers,

Richard

Re: [OE-core] [PATCH 2/2] recipetool: Handle several go-import tags in go resolver

[Sven Schwermer]

Hi,

On 4/11/24 2:17 PM, Richard Purdie wrote:
> 
> Should we be extending "oe-selftest -r recipetool" with tests for something?
> 

Good question, I would have to look into the self-testing framework. I 
don't have experience with that yet. Would you consider adding a test a 
requirement to get this merged?

Sven