[PATCH 0/3] Fix some bitbake authentication issues

[PATCH 0/3] Fix some bitbake authentication issues

[Stefan Klug]

Hi,

while using bitbake in our corporate environment I stumbled over some 
authentication/stability issues.
This is my first patchset to this list. So please correct me if I
missed anything.

Regards Stefan

Stefan Klug (3):
  fetch2: Fix fetching of git repositories with kerberos authentication
  fetch2: Gracefully handle corrupt download-cache tarballs
  fetch2/wget: Fix authentication in checkstatus() of the wget fetcher

 lib/bb/fetch2/__init__.py | 1 +
 lib/bb/fetch2/git.py      | 7 +++++--
 lib/bb/fetch2/wget.py     | 4 ++--
 3 files changed, 8 insertions(+), 4 deletions(-)

-- 

[PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication

[Stefan Klug]

When using pam_krb to login to a system KRB5CCNAME is set to the
corresponding kerberos auth cache file. The bitbake fetcher removes
this variable from the environment leading to a git authentication failure.
Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE
variables and relies on a hardcoded list.
Therefore it is impossible to fix this issue outside of bitbake.

Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
---
 lib/bb/fetch2/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index f112067d..df8e83e7 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
                   'GIT_SSH',
                   'GIT_SSL_CAINFO',
                   'GIT_SMART_HTTP',
+                  'KRB5CCNAME',
                   'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
                   'SOCKS5_USER', 'SOCKS5_PASSWD',
                   'DBUS_SESSION_BUS_ADDRESS',
-- 

[PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs

[Stefan Klug]

If the fullmirror tarball is corrupt for whatever reason
(IMHO there are no checksums on the download cache)
a series of nasty events was triggered:
- tar left a partially extracted bare git repo there
- on the next yocto build, the corrupt bare repo is
  found and bitbake starts to update that bare repo using git
- git fails to detect it as bare repo. Therefore
  all following git commands ripple up the directory tree,
  in our case modifying a top level git repo.

Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
---
 lib/bb/fetch2/git.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 1a8ebe3d..d5f9bbcd 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -334,8 +334,11 @@ class Git(FetchMethod):
             ud.localpath = ud.fullshallow
             return
         elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir):
-            bb.utils.mkdirhier(ud.clonedir)
-            runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir)
+            try:
+                bb.utils.mkdirhier(ud.clonedir)
+                runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir])
+            except:
+                logger.info("Extracting tarball of git repository failed, falling back to clone.")
 
         repourl = self._get_repo_url(ud)
 
-- 

[PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher

[Stefan Klug]

I wonder how this used to work for anybody.

Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
---
 lib/bb/fetch2/wget.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e2037511..3addb219 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -322,8 +322,8 @@ class Wget(FetchMethod):
                 authheader =  "Basic %s" % encodeuser
                 r.add_header("Authorization", authheader)
 
-            if ud.user:
-                add_basic_auth(ud.user, r)
+            if ud.user and ud.pswd:
+                add_basic_auth(ud.user + ':' + ud.pswd, r)
 
             try:
                 import netrc, urllib.parse
-- 

Re: [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication

[Richard Purdie]

On Mon, 2019-03-18 at 14:58 +0100, Stefan Klug wrote:
> When using pam_krb to login to a system KRB5CCNAME is set to the
> corresponding kerberos auth cache file. The bitbake fetcher removes
> this variable from the environment leading to a git authentication
> failure.
> Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE
> variables and relies on a hardcoded list.

This code should probably include the BB_ENV_[EXTRA_]WHITE list
variables...

That doesn't invalidate this patch but does suggest we have a bigger
problem.

Cheers,

Richard

> Therefore it is impossible to fix this issue outside of bitbake.
> 
> Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
> ---
>  lib/bb/fetch2/__init__.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index f112067d..df8e83e7 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False,
> cleanup=None, log=None, workdir=None):
>                    'GIT_SSH',
>                    'GIT_SSL_CAINFO',
>                    'GIT_SMART_HTTP',
> +                  'KRB5CCNAME',
>                    'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
>                    'SOCKS5_USER', 'SOCKS5_PASSWD',
>                    'DBUS_SESSION_BUS_ADDRESS',
> -- 
> 
> 

Re: [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs

[Richard Purdie]

On Mon, 2019-03-18 at 14:58 +0100, Stefan Klug wrote:
> If the fullmirror tarball is corrupt for whatever reason
> (IMHO there are no checksums on the download cache)
> a series of nasty events was triggered:
> - tar left a partially extracted bare git repo there
> - on the next yocto build, the corrupt bare repo is
>   found and bitbake starts to update that bare repo using git
> - git fails to detect it as bare repo. Therefore
>   all following git commands ripple up the directory tree,
>   in our case modifying a top level git repo.
> 
> Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
> ---
>  lib/bb/fetch2/git.py | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index 1a8ebe3d..d5f9bbcd 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -334,8 +334,11 @@ class Git(FetchMethod):
>              ud.localpath = ud.fullshallow
>              return
>          elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir):
> -            bb.utils.mkdirhier(ud.clonedir)
> -            runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir)
> +            try:
> +                bb.utils.mkdirhier(ud.clonedir)
> +                runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir])
> +            except:
> +                logger.info("Extracting tarball of git repository failed, falling back to clone.")

General "except:" clauses are a world of pain. Can we be more specific
here?

For an example of what I mean, put a syntax error in the command...

Cheers,

Richard

Re: [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher

[Andre McCurdy]

On Mon, Mar 18, 2019 at 7:08 AM Stefan Klug <stefan.klug@baslerweb.com> wrote:
>
> I wonder how this used to work for anybody.

No need to wonder when you have access to the git history... :-)

  http://git.openembedded.org/bitbake/commit/?id=cea8113d14da9e12db80a5b6b5811a47a7dfdeef

It looks like ud.user used to contain both the username and
password... and when that changed, download() was updated but
checkstatus() was not:

  http://git.openembedded.org/bitbake/commit/?id=6a917ec99d659e684b15fa8af94c325172676062

> Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
> ---
>  lib/bb/fetch2/wget.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
> index e2037511..3addb219 100644
> --- a/lib/bb/fetch2/wget.py
> +++ b/lib/bb/fetch2/wget.py
> @@ -322,8 +322,8 @@ class Wget(FetchMethod):
>                  authheader =  "Basic %s" % encodeuser
>                  r.add_header("Authorization", authheader)
>
> -            if ud.user:
> -                add_basic_auth(ud.user, r)
> +            if ud.user and ud.pswd:
> +                add_basic_auth(ud.user + ':' + ud.pswd, r)
>
>              try:
>                  import netrc, urllib.parse
> --
>
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel