From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yb0-f194.google.com (mail-yb0-f194.google.com [209.85.213.194]) by mail.openembedded.org (Postfix) with ESMTP id 9E55C7875A for ; Mon, 14 May 2018 18:52:30 +0000 (UTC) Received: by mail-yb0-f194.google.com with SMTP id w14-v6so4487720ybm.5 for ; Mon, 14 May 2018 11:52:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=E0eIuCPvz/h7odGn8I3j2VDR7wYcB5Q7cD8oGxu0LWw=; b=RUI5ISKrEoc1tdmNOtEYlfH0F0Gjq3r3TYjDqpTShN+Bssu2nnmNe9rL4lftIuD1w9 gkdKUwHhvQWlEdlOZv4bGSwar4/UbNPUAFWJlp9p5hYBAz1Li3kt0u9t3J6aYVCcKi1j JRUZtRZJenOQFcQ5HJsmVAhOFgPYKLHe0bQ5AfFNuw3mfQdKnXaKDo7+PZo+VHgCwF6H B4VipXQlEVtKAlD6Vx/7DyploYS226xd0dCJs0hR4OaJpp/osxwBYbXDt7ho1QsLKCAU g6ScY7YGRAZNpPUnDz9k2QPInuOspO1zY0jcx6dFDkAdvjjsqLbhfcS8vARnPobA6q1y 5W0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=E0eIuCPvz/h7odGn8I3j2VDR7wYcB5Q7cD8oGxu0LWw=; b=f1A1TH4fSoTZ2JRz2Ao1mPjNrIkIX/H0l8PGE6ppEWyGcRE7l3XbzASQG3ANaWHVGC XFOlQMsnAwckbn7m9dwxmwZ3/ClzIK85ATvu3pLKQ+RzIecPCyBWA7B9Ul2iEdGB0o+C QODT04wBNl2lxjjrBNbs1gQKPjztwPDph6hgg2nYYLURdzl5XjGJ/qW1wS8ilYLq6A/l j7nsPy9i1QoJ1zyTuekE8seJNT9hZabcbCyc135vzYDFfTAvSg0103RILyMR4ky2iVoj LkNzctJ3u2w92twQ8Kq+niFHiMxV4E2/ERafJbEvdvZH8eKr6TDw5WKP1a0GQ9Sc3o9S Z21A== X-Gm-Message-State: ALKqPweWW5DjVtsvJq4D9QWsVMnfzURk7Ff7PHmBf9Uiv4FQApXgVnIz g9MRK07UBOGOFXpbFzbCZjQPm+w4rhoyRqmJtYkMbA== X-Google-Smtp-Source: AB8JxZp1U7YBCewZSaKoX9hB8wZTl9pSuqRdE1W+QS00EodNGDQwUstDajTdSt27+77Ghtn+8pi2RzoBRG/074tBHw8= X-Received: by 2002:a25:b3cc:: with SMTP id x12-v6mr7350347ybf.181.1526323951368; Mon, 14 May 2018 11:52:31 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a25:748c:0:0:0:0:0 with HTTP; Mon, 14 May 2018 11:51:51 -0700 (PDT) In-Reply-To: <0ed6e7841d4949a98e6978944adf8c8e@XBOX02.axis.com> References: <20180511192636.6231-1-matt.hoosier@gmail.com> <0ed6e7841d4949a98e6978944adf8c8e@XBOX02.axis.com> From: Matt Hoosier Date: Mon, 14 May 2018 13:51:51 -0500 Message-ID: To: Peter Kjellerstedt Cc: "bitbake-devel@lists.openembedded.org" Subject: Re: [PATCH] fetch/gitsm: avoid live submodule fetching during unpack() X-BeenThere: bitbake-devel@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussion that advance bitbake development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2018 18:52:30 -0000 Content-Type: multipart/alternative; boundary="000000000000ef01ed056c2efcf4" --000000000000ef01ed056c2efcf4 Content-Type: text/plain; charset="UTF-8" On Mon, May 14, 2018 at 8:18 AM, Peter Kjellerstedt < peter.kjellerstedt@axis.com> wrote: > > -----Original Message----- > > From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake- > > devel-bounces@lists.openembedded.org] On Behalf Of Matt Hoosier > > Sent: den 11 maj 2018 21:27 > > To: bitbake-devel@lists.openembedded.org > > Subject: [bitbake-devel] [PATCH] fetch/gitsm: avoid live submodule > > fetching during unpack() > > > > Although the submodules' histories have been fetched during the > > do_fetch() phase, the mechanics used to clone the workdir copy > > of the repo haven't been transferring the actual .git/modules > > directory from the repo fetched into downloads/ during the > > fetch task. > > > > Fix that, and for good measure also explicitly tell Git to avoid > > hitting the network during do_unpack() of the submodules. > > > > Rev3 changes: > > > > - Really fix the shallow-clone issue claimed to be addressed in rev2. > > > > Rev2 changes: > > > > - Fixed a failure when unpacking a shallow cloned mirror repository. > > The Rev2 and Rev3 changes information should go below the --- below > as it should not be part of the commit message as it relates to the > patch submission process rather than the actual change. If any of it > is relevant for the commit message, then the commit message should > be changed as well. > Thanks; done. Resubmitted (as v4) to the list. > > > [YOCTO #12739] > > > > Signed-off-by: Matt Hoosier > > --- > > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- > > 1 file changed, 27 insertions(+), 1 deletion(-) > > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > > index 0aff1008..3969f2f5 100644 > > --- a/lib/bb/fetch2/gitsm.py > > +++ b/lib/bb/fetch2/gitsm.py > > @@ -132,4 +132,30 @@ class GitSM(Git): > > > > if self.uses_submodules(ud, d, ud.destdir): > > runfetchcmd(ud.basecmd + " checkout " + > ud.revisions[ud.names[0]], d, workdir=ud.destdir) > > - runfetchcmd(ud.basecmd + " submodule update --init > --recursive", d, workdir=ud.destdir) > > + > > + updateflags = "" > > + > > + # Copy over the submodules' fetched histories too. > > + if ud.bareclone: > > + repo_conf = ud.destdir > > + else: > > + repo_conf = os.path.join(ud.destdir, '.git') > > + > > + if os.path.exists(ud.clonedir): > > + # This is not a copy unpacked from a shallow mirror > clone. So > > + # the manual intervention to populate the .git/modules > done > > + # in clone_shallow_local() won't have been done yet. > > + runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, > 'modules'), repo_conf), d) > > + updateflags += " --no-fetch" > > + elif os.path.exists(os.path.join(repo_conf, 'modules')): > > + # Unpacked from a shallow mirror clone. Manual > population of > > + # .git/modules is already done. > > + updateflags += " --no-fetch" > > + else: > > + # This isn't fatal; git-submodule will just fetch it > > + # during do_unpack(). > > + bb.error("submodule history not retrieved during > do_fetch()") > > + > > + # Careful not to hit the network during unpacking; all > history should already > > + # be fetched. > > + runfetchcmd(ud.basecmd + " submodule update --init > --recursive %s" % updateflags, d, workdir=ud.destdir) > > -- > > 2.13.6 > > //Peter > > --000000000000ef01ed056c2efcf4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Mon, May 14, 2018 at 8:18 AM, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote:
> -----Original Message-----
> From: = bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@= lists.openembedded.org] On Behalf Of Matt Hoosier
> Sent: den 11 maj 2018 21:27
> To: bitbake-de= vel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH] fetch/gitsm: avoid live submodule
> fetching during unpack()
>
> Although the submodules' histories have been fetched during the > do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
>
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
>
> Rev3 changes:
>
> - Really fix the shallow-clone issue claimed to be addressed in rev2.<= br> >
> Rev2 changes:
>
> - Fixed a failure when unpacking a shallow cloned mirror repository.
The Rev2 and Rev3 changes information should go below the --- below =
as it should not be part of the commit message as it relates to the
patch submission process rather than the actual change. If any of it
is relevant for the commit message, then the commit message should
be changed as well.

Thanks; done. Resub= mitted (as v4) to the list.
=C2=A0

> [YOCTO #12739]
>
> Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> ---
>=C2=A0 lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
>=C2=A0 1 file changed, 27 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..3969f2f5 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,30 @@ class GitSM(Git):
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if self.uses_submodules(ud, d, ud.de= stdir):
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(ud.basecmd= + " checkout " + ud.revisions[ud.names[0]], d, workdir=3Dud.dest= dir)
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(ud.basecmd + &q= uot; submodule update --init --recursive", d, workdir=3Dud.destdir) > +
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 updateflags =3D ""= ;
> +
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Copy over the submodules&= #39; fetched histories too.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ud.bareclone:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 repo_conf =3D= ud.destdir
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 repo_conf =3D= os.path.join(ud.destdir, '.git')
> +
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if os.path.exists(ud.cloned= ir):
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # This is not= a copy unpacked from a shallow mirror clone. So
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # the manual = intervention to populate the .git/modules done
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # in clone_sh= allow_local() won't have been done yet.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(&= quot;cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), re= po_conf), d)
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 updateflags += =3D " --no-fetch"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 elif os.path.exists(os.path= .join(repo_conf, 'modules')):
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Unpacked fr= om a shallow mirror clone. Manual population of
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # .git/module= s is already done.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 updateflags += =3D " --no-fetch"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # This isn= 9;t fatal; git-submodule will just fetch it
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # during do_u= npack().
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bb.error(&quo= t;submodule history not retrieved during do_fetch()")
> +
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Careful not to hit the ne= twork during unpacking; all history should already
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # be fetched.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(ud.basecmd + &q= uot; submodule update --init --recursive %s" % updateflags, d, workdir= =3Dud.destdir)
> --
> 2.13.6

//Peter


--000000000000ef01ed056c2efcf4--