* [PATCH 0/3] Fix some bitbake authentication issues
@ 2019-03-18 13:58 Stefan Klug
2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Stefan Klug @ 2019-03-18 13:58 UTC (permalink / raw)
To: bitbake-devel
Hi,
while using bitbake in our corporate environment I stumbled over some
authentication/stability issues.
This is my first patchset to this list. So please correct me if I
missed anything.
Regards Stefan
Stefan Klug (3):
fetch2: Fix fetching of git repositories with kerberos authentication
fetch2: Gracefully handle corrupt download-cache tarballs
fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
lib/bb/fetch2/__init__.py | 1 +
lib/bb/fetch2/git.py | 7 +++++--
lib/bb/fetch2/wget.py | 4 ++--
3 files changed, 8 insertions(+), 4 deletions(-)
--
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication
2019-03-18 13:58 [PATCH 0/3] Fix some bitbake authentication issues Stefan Klug
@ 2019-03-18 13:58 ` Stefan Klug
2019-03-18 23:44 ` Richard Purdie
2019-03-18 13:58 ` [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs Stefan Klug
2019-03-18 13:58 ` [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher Stefan Klug
2 siblings, 1 reply; 7+ messages in thread
From: Stefan Klug @ 2019-03-18 13:58 UTC (permalink / raw)
To: bitbake-devel
When using pam_krb to login to a system KRB5CCNAME is set to the
corresponding kerberos auth cache file. The bitbake fetcher removes
this variable from the environment leading to a git authentication failure.
Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE
variables and relies on a hardcoded list.
Therefore it is impossible to fix this issue outside of bitbake.
Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
---
lib/bb/fetch2/__init__.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index f112067d..df8e83e7 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
'GIT_SSH',
'GIT_SSL_CAINFO',
'GIT_SMART_HTTP',
+ 'KRB5CCNAME',
'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
'SOCKS5_USER', 'SOCKS5_PASSWD',
'DBUS_SESSION_BUS_ADDRESS',
--
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs
2019-03-18 13:58 [PATCH 0/3] Fix some bitbake authentication issues Stefan Klug
2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug
@ 2019-03-18 13:58 ` Stefan Klug
2019-03-18 23:45 ` Richard Purdie
2019-03-18 13:58 ` [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher Stefan Klug
2 siblings, 1 reply; 7+ messages in thread
From: Stefan Klug @ 2019-03-18 13:58 UTC (permalink / raw)
To: bitbake-devel
If the fullmirror tarball is corrupt for whatever reason
(IMHO there are no checksums on the download cache)
a series of nasty events was triggered:
- tar left a partially extracted bare git repo there
- on the next yocto build, the corrupt bare repo is
found and bitbake starts to update that bare repo using git
- git fails to detect it as bare repo. Therefore
all following git commands ripple up the directory tree,
in our case modifying a top level git repo.
Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
---
lib/bb/fetch2/git.py | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 1a8ebe3d..d5f9bbcd 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -334,8 +334,11 @@ class Git(FetchMethod):
ud.localpath = ud.fullshallow
return
elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir):
- bb.utils.mkdirhier(ud.clonedir)
- runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir)
+ try:
+ bb.utils.mkdirhier(ud.clonedir)
+ runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir])
+ except:
+ logger.info("Extracting tarball of git repository failed, falling back to clone.")
repourl = self._get_repo_url(ud)
--
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
2019-03-18 13:58 [PATCH 0/3] Fix some bitbake authentication issues Stefan Klug
2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug
2019-03-18 13:58 ` [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs Stefan Klug
@ 2019-03-18 13:58 ` Stefan Klug
2019-03-19 6:28 ` Andre McCurdy
2 siblings, 1 reply; 7+ messages in thread
From: Stefan Klug @ 2019-03-18 13:58 UTC (permalink / raw)
To: bitbake-devel
I wonder how this used to work for anybody.
Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
---
lib/bb/fetch2/wget.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e2037511..3addb219 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -322,8 +322,8 @@ class Wget(FetchMethod):
authheader = "Basic %s" % encodeuser
r.add_header("Authorization", authheader)
- if ud.user:
- add_basic_auth(ud.user, r)
+ if ud.user and ud.pswd:
+ add_basic_auth(ud.user + ':' + ud.pswd, r)
try:
import netrc, urllib.parse
--
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication
2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug
@ 2019-03-18 23:44 ` Richard Purdie
0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2019-03-18 23:44 UTC (permalink / raw)
To: Stefan Klug, bitbake-devel
On Mon, 2019-03-18 at 14:58 +0100, Stefan Klug wrote:
> When using pam_krb to login to a system KRB5CCNAME is set to the
> corresponding kerberos auth cache file. The bitbake fetcher removes
> this variable from the environment leading to a git authentication
> failure.
> Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE
> variables and relies on a hardcoded list.
This code should probably include the BB_ENV_[EXTRA_]WHITE list
variables...
That doesn't invalidate this patch but does suggest we have a bigger
problem.
Cheers,
Richard
> Therefore it is impossible to fix this issue outside of bitbake.
>
> Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
> ---
> lib/bb/fetch2/__init__.py | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index f112067d..df8e83e7 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False,
> cleanup=None, log=None, workdir=None):
> 'GIT_SSH',
> 'GIT_SSL_CAINFO',
> 'GIT_SMART_HTTP',
> + 'KRB5CCNAME',
> 'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
> 'SOCKS5_USER', 'SOCKS5_PASSWD',
> 'DBUS_SESSION_BUS_ADDRESS',
> --
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs
2019-03-18 13:58 ` [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs Stefan Klug
@ 2019-03-18 23:45 ` Richard Purdie
0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2019-03-18 23:45 UTC (permalink / raw)
To: Stefan Klug, bitbake-devel
On Mon, 2019-03-18 at 14:58 +0100, Stefan Klug wrote:
> If the fullmirror tarball is corrupt for whatever reason
> (IMHO there are no checksums on the download cache)
> a series of nasty events was triggered:
> - tar left a partially extracted bare git repo there
> - on the next yocto build, the corrupt bare repo is
> found and bitbake starts to update that bare repo using git
> - git fails to detect it as bare repo. Therefore
> all following git commands ripple up the directory tree,
> in our case modifying a top level git repo.
>
> Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
> ---
> lib/bb/fetch2/git.py | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index 1a8ebe3d..d5f9bbcd 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -334,8 +334,11 @@ class Git(FetchMethod):
> ud.localpath = ud.fullshallow
> return
> elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir):
> - bb.utils.mkdirhier(ud.clonedir)
> - runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir)
> + try:
> + bb.utils.mkdirhier(ud.clonedir)
> + runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir])
> + except:
> + logger.info("Extracting tarball of git repository failed, falling back to clone.")
General "except:" clauses are a world of pain. Can we be more specific
here?
For an example of what I mean, put a syntax error in the command...
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
2019-03-18 13:58 ` [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher Stefan Klug
@ 2019-03-19 6:28 ` Andre McCurdy
0 siblings, 0 replies; 7+ messages in thread
From: Andre McCurdy @ 2019-03-19 6:28 UTC (permalink / raw)
To: Stefan Klug; +Cc: bitbake-devel
On Mon, Mar 18, 2019 at 7:08 AM Stefan Klug <stefan.klug@baslerweb.com> wrote:
>
> I wonder how this used to work for anybody.
No need to wonder when you have access to the git history... :-)
http://git.openembedded.org/bitbake/commit/?id=cea8113d14da9e12db80a5b6b5811a47a7dfdeef
It looks like ud.user used to contain both the username and
password... and when that changed, download() was updated but
checkstatus() was not:
http://git.openembedded.org/bitbake/commit/?id=6a917ec99d659e684b15fa8af94c325172676062
> Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
> ---
> lib/bb/fetch2/wget.py | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
> index e2037511..3addb219 100644
> --- a/lib/bb/fetch2/wget.py
> +++ b/lib/bb/fetch2/wget.py
> @@ -322,8 +322,8 @@ class Wget(FetchMethod):
> authheader = "Basic %s" % encodeuser
> r.add_header("Authorization", authheader)
>
> - if ud.user:
> - add_basic_auth(ud.user, r)
> + if ud.user and ud.pswd:
> + add_basic_auth(ud.user + ':' + ud.pswd, r)
>
> try:
> import netrc, urllib.parse
> --
>
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-03-19 6:28 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-18 13:58 [PATCH 0/3] Fix some bitbake authentication issues Stefan Klug
2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug
2019-03-18 23:44 ` Richard Purdie
2019-03-18 13:58 ` [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs Stefan Klug
2019-03-18 23:45 ` Richard Purdie
2019-03-18 13:58 ` [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher Stefan Klug
2019-03-19 6:28 ` Andre McCurdy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.