[RFC PATCH] ci: ensure that all jobs use a shallow clone

[RFC PATCH] ci: ensure that all jobs use a shallow clone

[Paolo Bonzini]

We are still using quite a bit of bandwidth to run CI jobs, even though
GitLab has switched to gitlab.com to fetch the sources.  This is in
part because we are handling submodules ourselves and therefore those
do not use shallow clones.

Observe GitLab's GIT_DEPTH environment variable in
scripts/git-submodule.sh so that CI systems can use a shallow clone
for submodules.  Set a depth of 3 for all CI systems, for both the main
repository (using whatever native configuration mechanism is there)
and the submodules.

Reported-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
        Completely untested, just to propose the idea.

 .cirrus.yml              | 1 +
 .gitlab-ci.yml           | 3 +++
 .shippable.yml           | 2 ++
 .travis.yml              | 2 ++
 scripts/git-submodule.sh | 2 +-
 5 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 6f2a958472..47c2467d04 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,5 +1,6 @@
 env:
-  CIRRUS_CLONE_DEPTH: 1
+  CIRRUS_CLONE_DEPTH: 3
+  GIT_DEPTH: 3
 
 freebsd_12_task:
   freebsd_instance:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 01c9e46410..ff19a9c313 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -7,6 +7,9 @@ stages:
   - build
   - test
 
+variables:
+  GIT_DEPTH: 3
+
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
diff --git a/.shippable.yml b/.shippable.yml
index 14350e6de8..f2d59b478e 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,9 +1,11 @@
 language: c
+depth: 3
 git:
    submodules: false
 env:
   global:
     - LC_ALL=C
+    - GIT_DEPTH=3
   matrix:
     - IMAGE=debian-amd64
       TARGET_LIST=x86_64-softmmu,x86_64-linux-user
diff --git a/.travis.yml b/.travis.yml
index f2a101936c..f2a785a42b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -80,9 +80,11 @@ env:
     - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime"
     - CCACHE_MAXSIZE=1G
     - G_MESSAGES_DEBUG=error
+    - GIT_DEPTH=3
 
 
 git:
+  depth: 3
   # we want to do this ourselves
   submodules: false
 
diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh
index 65ed877aef..2886087a12 100755
--- a/scripts/git-submodule.sh
+++ b/scripts/git-submodule.sh
@@ -75,7 +75,7 @@ update)
         exit 0
     fi
 
-    $GIT submodule update --init $modules 1>/dev/null
+    $GIT submodule update ${GIT_DEPTH:+--depth "$GIT_DEPTH"} --init $modules 1>/dev/null
     test $? -ne 0 && error "failed to update modules"
 
     $GIT submodule status $modules > "${substat}"
-- 
2.29.2

Re: [RFC PATCH] ci: ensure that all jobs use a shallow clone

[Daniel P. Berrangé]

On Thu, Jan 07, 2021 at 06:17:19PM +0100, Paolo Bonzini wrote:
> We are still using quite a bit of bandwidth to run CI jobs, even though
> GitLab has switched to gitlab.com to fetch the sources.  This is in
> part because we are handling submodules ourselves and therefore those
> do not use shallow clones.

I presume the impact is going to vary for GitLab vs the other
CI systems.  For gitlab, any bandwidth used should be entirely
within gitlab's own network infra since its pulling repos
from  gitlab.com/qemu-project

For Cirrus/Shippable/Travis  the bandwidth used would be more
of a potential performance problem.

> Observe GitLab's GIT_DEPTH environment variable in
> scripts/git-submodule.sh so that CI systems can use a shallow clone
> for submodules.  Set a depth of 3 for all CI systems, for both the main
> repository (using whatever native configuration mechanism is there)
> and the submodules.
> 
> Reported-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>         Completely untested, just to propose the idea.

The check-patch and check-dco jobs need the git history depth to be
long enough to cover the entire contents of the branch that is being
tested vs common ancestor against git master.

This could potentially be 100's of commits deep for some of the very
largest branches we see merged in qemu


Travis has issues with git cloning and concurrent pushes.

eg if you push branch A, it schedules a CI job. Then you
push branch B before jobs for A have started.

When the job for A starts, it will be unable to checkout
the commit for branch A's merge because the history will
be too shallow.


I'm not sure if the other CI systems have similar issues or
not.

> 
>  .cirrus.yml              | 1 +
>  .gitlab-ci.yml           | 3 +++
>  .shippable.yml           | 2 ++
>  .travis.yml              | 2 ++
>  scripts/git-submodule.sh | 2 +-
>  5 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/.cirrus.yml b/.cirrus.yml
> index 6f2a958472..47c2467d04 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -1,5 +1,6 @@
>  env:
> -  CIRRUS_CLONE_DEPTH: 1
> +  CIRRUS_CLONE_DEPTH: 3
> +  GIT_DEPTH: 3
>  
>  freebsd_12_task:
>    freebsd_instance:
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 01c9e46410..ff19a9c313 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -7,6 +7,9 @@ stages:
>    - build
>    - test
>  
> +variables:
> +  GIT_DEPTH: 3
> +
>  include:
>    - local: '/.gitlab-ci.d/edk2.yml'
>    - local: '/.gitlab-ci.d/opensbi.yml'
> diff --git a/.shippable.yml b/.shippable.yml
> index 14350e6de8..f2d59b478e 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -1,9 +1,11 @@
>  language: c
> +depth: 3
>  git:
>     submodules: false
>  env:
>    global:
>      - LC_ALL=C
> +    - GIT_DEPTH=3
>    matrix:
>      - IMAGE=debian-amd64
>        TARGET_LIST=x86_64-softmmu,x86_64-linux-user
> diff --git a/.travis.yml b/.travis.yml
> index f2a101936c..f2a785a42b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -80,9 +80,11 @@ env:
>      - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime"
>      - CCACHE_MAXSIZE=1G
>      - G_MESSAGES_DEBUG=error
> +    - GIT_DEPTH=3
>  
>  
>  git:
> +  depth: 3
>    # we want to do this ourselves
>    submodules: false
>  
> diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh
> index 65ed877aef..2886087a12 100755
> --- a/scripts/git-submodule.sh
> +++ b/scripts/git-submodule.sh
> @@ -75,7 +75,7 @@ update)
>          exit 0
>      fi
>  
> -    $GIT submodule update --init $modules 1>/dev/null
> +    $GIT submodule update ${GIT_DEPTH:+--depth "$GIT_DEPTH"} --init $modules 1>/dev/null
>      test $? -ne 0 && error "failed to update modules"
>  
>      $GIT submodule status $modules > "${substat}"
> -- 
> 2.29.2
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] ci: ensure that all jobs use a shallow clone

[Thomas Huth]

On 07/01/2021 19.28, Daniel P. Berrangé wrote:
[...]
> Travis has issues with git cloning and concurrent pushes.
> 
> eg if you push branch A, it schedules a CI job. Then you
> push branch B before jobs for A have started.
> 
> When the job for A starts, it will be unable to checkout
> the commit for branch A's merge because the history will
> be too shallow.

I think we have to get rid of travis.yml soon anyway, since they changed 
their business model, and soon it won't be possible anymore to use it for 
QEMU. travis-ci.org is getting decommissioned and on travis-ci.com you can 
only get free CI minutes for non-sponsored FOSS projects.
So let's simply not worry about Travis-CI anymore.

>>   .cirrus.yml              | 1 +
>>   .gitlab-ci.yml           | 3 +++
>>   .shippable.yml           | 2 ++

Maybe we could rather disable shippable now that we support the cross 
container builds on gitlab-ci, too?

  Thomas

Re: [RFC PATCH] ci: ensure that all jobs use a shallow clone

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

Il gio 7 gen 2021, 20:05 Thomas Huth <thuth@redhat.com> ha scritto:

> on travis-ci.com you can
> only get free CI minutes for non-sponsored FOSS projects.
> So let's simply not worry about Travis-CI anymore.
>
> Maybe we could rather disable shippable now that we support the cross
> container builds on gitlab-ci, too?
>

With pleasure, starting this discussion was an intended possible side
effect of the patch. :)

The main issue with Travis is the non-x86 builders, which have no
alternative yet.

Paolo


>   Thomas
>
>

[-- Attachment #2: Type: text/html, Size: 1283 bytes --]

Re: [RFC PATCH] ci: ensure that all jobs use a shallow clone

[Daniel P. Berrangé]

On Thu, Jan 07, 2021 at 08:23:49PM +0100, Paolo Bonzini wrote:
> Il gio 7 gen 2021, 20:05 Thomas Huth <thuth@redhat.com> ha scritto:
> 
> > on travis-ci.com you can
> > only get free CI minutes for non-sponsored FOSS projects.
> > So let's simply not worry about Travis-CI anymore.
> >
> > Maybe we could rather disable shippable now that we support the cross
> > container builds on gitlab-ci, too?
> >
> 
> With pleasure, starting this discussion was an intended possible side
> effect of the patch. :)
> 
> The main issue with Travis is the non-x86 builders, which have no
> alternative yet.

The free travis accounts get 10000 credits. Linux jobs take 10 credits
per minute of execution time. Free credits are a one-off grant which
don't auto renew. You have to make an email request to travis to beg
for more each time they run out.

The non-x86 CI jobs we have in travis consume 91 minutes of time in
total.  So we'll burn through all free credits in 11 jobs and then
our CI will stop.

IOW, despite travis giving us non-x86 builders, it is doomed to be
unusuable, unless we can convince them to give us a *massively*
larger free credit allowance on the qemu account.  It would need to
be on the order x100 larger, and auto-renewing once a month to cope
with our rate of builds for the non-x86 CI jobs.

I think we need to just delete the travis config and accept that we
can't run those jobs unless we provide our own hardware for non-x86
in GitLab CI.  IIUC, the latter is something we're planning anyway.

So I support removing travis and shippable configs and focusing only
on GitLab and Cirrus

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] ci: ensure that all jobs use a shallow clone

[Thomas Huth]

On 08/01/2021 11.16, Daniel P. Berrangé wrote:

> IOW, despite travis giving us non-x86 builders, it is doomed to be
> unusuable, unless we can convince them to give us a *massively*
> larger free credit allowance on the qemu account. 
I think convincing them to do this will be very hard. I've tried to apply to 
the free OSS credits for my account, and in the mail that I've got back 
after a couple of weeks, they wrote that "Project must not be sponsored by a 
commercial company or organization (monetary or with employees paid to work 
on the project)" to qualify for the free credits. That doesn't sound like 
they'll accept QEMU, I think.

  Thomas

Re: [RFC PATCH] ci: ensure that all jobs use a shallow clone

[Daniel P. Berrangé]

On Fri, Jan 08, 2021 at 12:06:01PM +0100, Thomas Huth wrote:
> On 08/01/2021 11.16, Daniel P. Berrangé wrote:
> 
> > IOW, despite travis giving us non-x86 builders, it is doomed to be
> > unusuable, unless we can convince them to give us a *massively*
> > larger free credit allowance on the qemu account.
> I think convincing them to do this will be very hard. I've tried to apply to
> the free OSS credits for my account, and in the mail that I've got back
> after a couple of weeks, they wrote that "Project must not be sponsored by a
> commercial company or organization (monetary or with employees paid to work
> on the project)" to qualify for the free credits. That doesn't sound like
> they'll accept QEMU, I think.

Yeah that makes their OSS program unviable for any project which isn't
essentially just a single person free time effort.

Lets drop Travis and not waste any more time on this.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|