From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBF23C433DB for ; Fri, 19 Feb 2021 08:00:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5443B64E86 for ; Fri, 19 Feb 2021 08:00:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5443B64E86 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:48544 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lD0hr-0005XJ-RD for qemu-devel@archiver.kernel.org; Fri, 19 Feb 2021 03:00:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:52312) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lD0fe-0003L1-9x for qemu-devel@nongnu.org; Fri, 19 Feb 2021 02:57:50 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:42492) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1lD0fc-0002Yt-KM for qemu-devel@nongnu.org; Fri, 19 Feb 2021 02:57:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1613721467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DBuXV/M7l4r2DQi+yhhAfjrJHFwjkkIeFU9NjSOcn68=; b=J++X5dupXqVnGrjwVhjqvlb73be/JeokTsllmKZlyau93R3PtiDhgThX03XcFvtI7oUW+S teHHMUOtLimFsjXAuwdP2vpv/7sfRmMk55Pf5VdhDylhm0rJ3IvLckMd3c1iI70ZviqEB4 TcdfLuv8i8ErCMP/7Hk+OKc69QIzPR0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-434-3HHllDWAMyaxhVXpG4T3Pw-1; Fri, 19 Feb 2021 02:57:45 -0500 X-MC-Unique: 3HHllDWAMyaxhVXpG4T3Pw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B06AF1005501; Fri, 19 Feb 2021 07:57:44 +0000 (UTC) Received: from thuth.com (ovpn-112-63.ams2.redhat.com [10.36.112.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id 688CF5D9C2; Fri, 19 Feb 2021 07:57:43 +0000 (UTC) From: Thomas Huth To: qemu-devel@nongnu.org, Peter Maydell Subject: [PULL 02/10] gitlab: always build container images Date: Fri, 19 Feb 2021 08:57:30 +0100 Message-Id: <20210219075738.2261103-3-thuth@redhat.com> In-Reply-To: <20210219075738.2261103-1-thuth@redhat.com> References: <20210219075738.2261103-1-thuth@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=thuth@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=63.128.21.124; envelope-from=thuth@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , =?UTF-8?q?Alex=20Benn=C3=A9e?= Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Daniel P. Berrangé Currently we attempt to skip building container images if the commits do not involve changes to the dockerfiles or gitlab CI definitions. Conceptually this makes sense, but there is a challenge in the real world implementation of this in gitlab. In the case of a CI pipeline triggered from a merge request, GitLab knows the common ancestor of the merge request and the main git repo, so it can trivially determine if any of the commits associated with the MR change the dockerfiles. In the case of a CI pipeline triggered from a push to a branch, it is much more difficult. There is no concept of a common ancestor in this case. Instead GitLab looks at the set of commits in the git push event. On the surface this may sound reasonable, but it doesn't take into account that a push event does not always contain the full set of patches from a branch. For example, consider pushing 5 commits, one of which contains a dockerfile change. This will trigger a CI pipeline for the containers. Now consider you do some more work on the branch and push 3 further commits, so you now have a branch of 8 commits. For the second push GitLab will only look at the 3 most recent commits, the other 5 were already present. Thus GitLab will not realize that the branch has dockerfile changes that need to trigger the container build. This can cause real world problems: - Push 5 commits to branch "foo", including a dockerfile change => rebuilds the container images with content from "foo" => build jobs runs against containers from "foo" - Refresh your master branch with latest upstream master => rebuilds the container images with content from "master" => build jobs runs against containers from "master" - Push 3 more commits to branch "foo", with no dockerfile change => no container rebuild triggers => build jobs runs against containers from "master" The "changes" conditional in gitlab is OK, *provided* your build jobs are not relying on any external state from previous builds. This is NOT the case in QEMU, because we are building container images and these are cached. This is a scenario in which the "changes" conditional is not usuable. The only other way to avoid this problem would be to use the git branch name as the container image tag, instead of always using "latest". The downside of this approach is that the user's gitlab registry will grow significantly until it starts to trigger GitLab's automatic deletion policy. Every time the user starts a new branch they will have to trigger a rebuild of the container images. Given this, we might as well just drop the conditional and always build the container images. Most of the time docker will be able to use the layer cache to avoid the most expensive part of the rebuild process (installing all the RPMs/debs/etc) Signed-off-by: Daniel P. Berrangé Message-Id: <20210216132954.295906-2-berrange@redhat.com> Signed-off-by: Thomas Huth --- .gitlab-ci.d/containers.yml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml index 90fac85ce4..33e4046e23 100644 --- a/.gitlab-ci.d/containers.yml +++ b/.gitlab-ci.d/containers.yml @@ -20,13 +20,6 @@ - docker push "$TAG" after_script: - docker logout - rules: - - changes: - - .gitlab-ci.d/containers.yml - - tests/docker/* - - tests/docker/dockerfiles/* - - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' - - if: '$CI_COMMIT_REF_NAME == "testing/next"' amd64-alpine-container: <<: *container_job_definition -- 2.27.0