* [PATCH 0/3] Fix some bitbake authentication issues @ 2019-03-18 13:58 Stefan Klug 2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Stefan Klug @ 2019-03-18 13:58 UTC (permalink / raw) To: bitbake-devel Hi, while using bitbake in our corporate environment I stumbled over some authentication/stability issues. This is my first patchset to this list. So please correct me if I missed anything. Regards Stefan Stefan Klug (3): fetch2: Fix fetching of git repositories with kerberos authentication fetch2: Gracefully handle corrupt download-cache tarballs fetch2/wget: Fix authentication in checkstatus() of the wget fetcher lib/bb/fetch2/__init__.py | 1 + lib/bb/fetch2/git.py | 7 +++++-- lib/bb/fetch2/wget.py | 4 ++-- 3 files changed, 8 insertions(+), 4 deletions(-) -- ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication 2019-03-18 13:58 [PATCH 0/3] Fix some bitbake authentication issues Stefan Klug @ 2019-03-18 13:58 ` Stefan Klug 2019-03-18 23:44 ` Richard Purdie 2019-03-18 13:58 ` [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs Stefan Klug 2019-03-18 13:58 ` [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher Stefan Klug 2 siblings, 1 reply; 7+ messages in thread From: Stefan Klug @ 2019-03-18 13:58 UTC (permalink / raw) To: bitbake-devel When using pam_krb to login to a system KRB5CCNAME is set to the corresponding kerberos auth cache file. The bitbake fetcher removes this variable from the environment leading to a git authentication failure. Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE variables and relies on a hardcoded list. Therefore it is impossible to fix this issue outside of bitbake. Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> --- lib/bb/fetch2/__init__.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index f112067d..df8e83e7 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None): 'GIT_SSH', 'GIT_SSL_CAINFO', 'GIT_SMART_HTTP', + 'KRB5CCNAME', 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'SOCKS5_USER', 'SOCKS5_PASSWD', 'DBUS_SESSION_BUS_ADDRESS', -- ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication 2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug @ 2019-03-18 23:44 ` Richard Purdie 0 siblings, 0 replies; 7+ messages in thread From: Richard Purdie @ 2019-03-18 23:44 UTC (permalink / raw) To: Stefan Klug, bitbake-devel On Mon, 2019-03-18 at 14:58 +0100, Stefan Klug wrote: > When using pam_krb to login to a system KRB5CCNAME is set to the > corresponding kerberos auth cache file. The bitbake fetcher removes > this variable from the environment leading to a git authentication > failure. > Also the fetcher ignores the normally used BB_ENV_[EXTRA_]WHITE > variables and relies on a hardcoded list. This code should probably include the BB_ENV_[EXTRA_]WHITE list variables... That doesn't invalidate this patch but does suggest we have a bigger problem. Cheers, Richard > Therefore it is impossible to fix this issue outside of bitbake. > > Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> > --- > lib/bb/fetch2/__init__.py | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py > index f112067d..df8e83e7 100644 > --- a/lib/bb/fetch2/__init__.py > +++ b/lib/bb/fetch2/__init__.py > @@ -830,6 +830,7 @@ def runfetchcmd(cmd, d, quiet=False, > cleanup=None, log=None, workdir=None): > 'GIT_SSH', > 'GIT_SSL_CAINFO', > 'GIT_SMART_HTTP', > + 'KRB5CCNAME', > 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', > 'SOCKS5_USER', 'SOCKS5_PASSWD', > 'DBUS_SESSION_BUS_ADDRESS', > -- > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs 2019-03-18 13:58 [PATCH 0/3] Fix some bitbake authentication issues Stefan Klug 2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug @ 2019-03-18 13:58 ` Stefan Klug 2019-03-18 23:45 ` Richard Purdie 2019-03-18 13:58 ` [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher Stefan Klug 2 siblings, 1 reply; 7+ messages in thread From: Stefan Klug @ 2019-03-18 13:58 UTC (permalink / raw) To: bitbake-devel If the fullmirror tarball is corrupt for whatever reason (IMHO there are no checksums on the download cache) a series of nasty events was triggered: - tar left a partially extracted bare git repo there - on the next yocto build, the corrupt bare repo is found and bitbake starts to update that bare repo using git - git fails to detect it as bare repo. Therefore all following git commands ripple up the directory tree, in our case modifying a top level git repo. Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> --- lib/bb/fetch2/git.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index 1a8ebe3d..d5f9bbcd 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -334,8 +334,11 @@ class Git(FetchMethod): ud.localpath = ud.fullshallow return elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir): - bb.utils.mkdirhier(ud.clonedir) - runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir) + try: + bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir]) + except: + logger.info("Extracting tarball of git repository failed, falling back to clone.") repourl = self._get_repo_url(ud) -- ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs 2019-03-18 13:58 ` [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs Stefan Klug @ 2019-03-18 23:45 ` Richard Purdie 0 siblings, 0 replies; 7+ messages in thread From: Richard Purdie @ 2019-03-18 23:45 UTC (permalink / raw) To: Stefan Klug, bitbake-devel On Mon, 2019-03-18 at 14:58 +0100, Stefan Klug wrote: > If the fullmirror tarball is corrupt for whatever reason > (IMHO there are no checksums on the download cache) > a series of nasty events was triggered: > - tar left a partially extracted bare git repo there > - on the next yocto build, the corrupt bare repo is > found and bitbake starts to update that bare repo using git > - git fails to detect it as bare repo. Therefore > all following git commands ripple up the directory tree, > in our case modifying a top level git repo. > > Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> > --- > lib/bb/fetch2/git.py | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) > > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py > index 1a8ebe3d..d5f9bbcd 100644 > --- a/lib/bb/fetch2/git.py > +++ b/lib/bb/fetch2/git.py > @@ -334,8 +334,11 @@ class Git(FetchMethod): > ud.localpath = ud.fullshallow > return > elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir): > - bb.utils.mkdirhier(ud.clonedir) > - runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir) > + try: > + bb.utils.mkdirhier(ud.clonedir) > + runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir, cleanup=[ud.clonedir]) > + except: > + logger.info("Extracting tarball of git repository failed, falling back to clone.") General "except:" clauses are a world of pain. Can we be more specific here? For an example of what I mean, put a syntax error in the command... Cheers, Richard ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher 2019-03-18 13:58 [PATCH 0/3] Fix some bitbake authentication issues Stefan Klug 2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug 2019-03-18 13:58 ` [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs Stefan Klug @ 2019-03-18 13:58 ` Stefan Klug 2019-03-19 6:28 ` Andre McCurdy 2 siblings, 1 reply; 7+ messages in thread From: Stefan Klug @ 2019-03-18 13:58 UTC (permalink / raw) To: bitbake-devel I wonder how this used to work for anybody. Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> --- lib/bb/fetch2/wget.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py index e2037511..3addb219 100644 --- a/lib/bb/fetch2/wget.py +++ b/lib/bb/fetch2/wget.py @@ -322,8 +322,8 @@ class Wget(FetchMethod): authheader = "Basic %s" % encodeuser r.add_header("Authorization", authheader) - if ud.user: - add_basic_auth(ud.user, r) + if ud.user and ud.pswd: + add_basic_auth(ud.user + ':' + ud.pswd, r) try: import netrc, urllib.parse -- ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher 2019-03-18 13:58 ` [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher Stefan Klug @ 2019-03-19 6:28 ` Andre McCurdy 0 siblings, 0 replies; 7+ messages in thread From: Andre McCurdy @ 2019-03-19 6:28 UTC (permalink / raw) To: Stefan Klug; +Cc: bitbake-devel On Mon, Mar 18, 2019 at 7:08 AM Stefan Klug <stefan.klug@baslerweb.com> wrote: > > I wonder how this used to work for anybody. No need to wonder when you have access to the git history... :-) http://git.openembedded.org/bitbake/commit/?id=cea8113d14da9e12db80a5b6b5811a47a7dfdeef It looks like ud.user used to contain both the username and password... and when that changed, download() was updated but checkstatus() was not: http://git.openembedded.org/bitbake/commit/?id=6a917ec99d659e684b15fa8af94c325172676062 > Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com> > --- > lib/bb/fetch2/wget.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py > index e2037511..3addb219 100644 > --- a/lib/bb/fetch2/wget.py > +++ b/lib/bb/fetch2/wget.py > @@ -322,8 +322,8 @@ class Wget(FetchMethod): > authheader = "Basic %s" % encodeuser > r.add_header("Authorization", authheader) > > - if ud.user: > - add_basic_auth(ud.user, r) > + if ud.user and ud.pswd: > + add_basic_auth(ud.user + ':' + ud.pswd, r) > > try: > import netrc, urllib.parse > -- > > > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-03-19 6:28 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-03-18 13:58 [PATCH 0/3] Fix some bitbake authentication issues Stefan Klug 2019-03-18 13:58 ` [PATCH 1/3] fetch2: Fix fetching of git repositories with kerberos authentication Stefan Klug 2019-03-18 23:44 ` Richard Purdie 2019-03-18 13:58 ` [PATCH 2/3] fetch2: Gracefully handle corrupt download-cache tarballs Stefan Klug 2019-03-18 23:45 ` Richard Purdie 2019-03-18 13:58 ` [PATCH 3/3] fetch2/wget: Fix authentication in checkstatus() of the wget fetcher Stefan Klug 2019-03-19 6:28 ` Andre McCurdy
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.