* gitlab containers are broken
@ 2021-02-03 23:04 Richard Henderson
2021-02-04 6:03 ` Thomas Huth
0 siblings, 1 reply; 10+ messages in thread
From: Richard Henderson @ 2021-02-03 23:04 UTC (permalink / raw)
To: qemu-devel, Thomas Huth, Philippe Mathieu-Daudé, Alex Bennée
Something has gone wrong with the building of the containers
in gitlab, because *all* off them are installing Alpine Linux.
https://gitlab.com/rth7680/qemu/-/jobs/1006336396#L155
I presume that IMAGE is not actually being passed through, and alpine.docker is
lexicographically first.
I have a strong suspicion that it's related to local "make docker" breakage, in
that e.g.
$ make docker-test-build@fedora-i386-cross
/usr/bin/python3 -B /home/rth/qemu/qemu/meson/meson.py introspect --targets
--tests --benchmarks | /usr/bin/python3 -B scripts/mtest2make.py > Makefile.mtest
GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 meson dtc capstone slirp
GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 meson dtc capstone slirp
make: *** No rule to make target 'docker-test-build@fedora-i386-cross'. Stop.
which certainly looks like the docker-TEST@IMAGE format documented.
r~
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-03 23:04 gitlab containers are broken Richard Henderson
@ 2021-02-04 6:03 ` Thomas Huth
2021-02-04 6:27 ` Richard Henderson
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Huth @ 2021-02-04 6:03 UTC (permalink / raw)
To: Richard Henderson, qemu-devel, Philippe Mathieu-Daudé,
Alex Bennée
On 04/02/2021 00.04, Richard Henderson wrote:
> Something has gone wrong with the building of the containers
> in gitlab, because *all* off them are installing Alpine Linux.
>
> https://gitlab.com/rth7680/qemu/-/jobs/1006336396#L155
I think that's ok ... the output about alpine that you see there is just the
output from the container that builds the final container. Later you can see
some "yum install" lines in that output, too, that's where the CentOS
container gets build. And the final compilation job runs on CentOS, too:
https://gitlab.com/rth7680/qemu/-/jobs/1006336699#L35
(look for the string "Red Hat" there)
> I presume that IMAGE is not actually being passed through, and alpine.docker is
> lexicographically first.
>
> I have a strong suspicion that it's related to local "make docker" breakage, in
> that e.g.
>
> $ make docker-test-build@fedora-i386-cross
> /usr/bin/python3 -B /home/rth/qemu/qemu/meson/meson.py introspect --targets
> --tests --benchmarks | /usr/bin/python3 -B scripts/mtest2make.py > Makefile.mtest
> GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3
> tests/fp/berkeley-softfloat-3 meson dtc capstone slirp
> GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3
> tests/fp/berkeley-softfloat-3 meson dtc capstone slirp
> make: *** No rule to make target 'docker-test-build@fedora-i386-cross'. Stop.
>
> which certainly looks like the docker-TEST@IMAGE format documented.
No clue about that, local containers never really worked for me... Alex?
Philippe? Any ideas?
Thomas
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-04 6:03 ` Thomas Huth
@ 2021-02-04 6:27 ` Richard Henderson
2021-02-04 8:08 ` Thomas Huth
0 siblings, 1 reply; 10+ messages in thread
From: Richard Henderson @ 2021-02-04 6:27 UTC (permalink / raw)
To: Thomas Huth, qemu-devel, Philippe Mathieu-Daudé, Alex Bennée
On 2/3/21 8:03 PM, Thomas Huth wrote:
> On 04/02/2021 00.04, Richard Henderson wrote:
>> Something has gone wrong with the building of the containers
>> in gitlab, because *all* off them are installing Alpine Linux.
>>
>> https://gitlab.com/rth7680/qemu/-/jobs/1006336396#L155
>
> I think that's ok ... the output about alpine that you see there is just the
> output from the container that builds the final container. Later you can see
> some "yum install" lines in that output, too, that's where the CentOS container
> gets build. And the final compilation job runs on CentOS, too:
>
> https://gitlab.com/rth7680/qemu/-/jobs/1006336699#L35
>
> (look for the string "Red Hat" there)
Hmm. Is there any way to get the full output of the container build? At
present it's being truncated:
#7 [4/5] RUN yum install -y bzip2 bzip2-devel ccache csnappy-de...
In particular, I'm trying to add a new test, and I have added libffi-devel.i686
to the fedora-i386-cross.docker file, but then the actual build fails because
the libffi header file is missing.
I know you may need the actual patch to comment, but pointers to how to debug
this sort of failure are welcome.
r~
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-04 6:27 ` Richard Henderson
@ 2021-02-04 8:08 ` Thomas Huth
2021-02-04 10:00 ` Daniel P. Berrangé
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Huth @ 2021-02-04 8:08 UTC (permalink / raw)
To: Richard Henderson, qemu-devel, Philippe Mathieu-Daudé,
Alex Bennée
Cc: Daniel P. Berrange
On 04/02/2021 07.27, Richard Henderson wrote:
> On 2/3/21 8:03 PM, Thomas Huth wrote:
>> On 04/02/2021 00.04, Richard Henderson wrote:
>>> Something has gone wrong with the building of the containers
>>> in gitlab, because *all* off them are installing Alpine Linux.
>>>
>>> https://gitlab.com/rth7680/qemu/-/jobs/1006336396#L155
>>
>> I think that's ok ... the output about alpine that you see there is just the
>> output from the container that builds the final container. Later you can see
>> some "yum install" lines in that output, too, that's where the CentOS container
>> gets build. And the final compilation job runs on CentOS, too:
>>
>> https://gitlab.com/rth7680/qemu/-/jobs/1006336699#L35
>>
>> (look for the string "Red Hat" there)
>
> Hmm. Is there any way to get the full output of the container build? At
> present it's being truncated:
>
> #7 [4/5] RUN yum install -y bzip2 bzip2-devel ccache csnappy-de...
>
>
> In particular, I'm trying to add a new test, and I have added libffi-devel.i686
> to the fedora-i386-cross.docker file, but then the actual build fails because
> the libffi header file is missing.
>
> I know you may need the actual patch to comment, but pointers to how to debug
> this sort of failure are welcome.
I don't have a clue, all that container magic has been done by Daniel
initially - maybe he can help (now on CC:) ...
Thomas
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-04 8:08 ` Thomas Huth
@ 2021-02-04 10:00 ` Daniel P. Berrangé
2021-02-04 17:36 ` Richard Henderson
0 siblings, 1 reply; 10+ messages in thread
From: Daniel P. Berrangé @ 2021-02-04 10:00 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, Philippe Mathieu-Daudé,
Richard Henderson, qemu-devel
On Thu, Feb 04, 2021 at 09:08:13AM +0100, Thomas Huth wrote:
> On 04/02/2021 07.27, Richard Henderson wrote:
> > On 2/3/21 8:03 PM, Thomas Huth wrote:
> > > On 04/02/2021 00.04, Richard Henderson wrote:
> > > > Something has gone wrong with the building of the containers
> > > > in gitlab, because *all* off them are installing Alpine Linux.
> > > >
> > > > https://gitlab.com/rth7680/qemu/-/jobs/1006336396#L155
> > >
> > > I think that's ok ... the output about alpine that you see there is just the
> > > output from the container that builds the final container. Later you can see
> > > some "yum install" lines in that output, too, that's where the CentOS container
> > > gets build. And the final compilation job runs on CentOS, too:
> > >
> > > https://gitlab.com/rth7680/qemu/-/jobs/1006336699#L35
> > >
> > > (look for the string "Red Hat" there)
> >
> > Hmm. Is there any way to get the full output of the container build? At
> > present it's being truncated:
> >
> > #7 [4/5] RUN yum install -y bzip2 bzip2-devel ccache csnappy-de...
> >
> >
> > In particular, I'm trying to add a new test, and I have added libffi-devel.i686
> > to the fedora-i386-cross.docker file, but then the actual build fails because
> > the libffi header file is missing.
> >
> > I know you may need the actual patch to comment, but pointers to how to debug
> > this sort of failure are welcome.
>
> I don't have a clue, all that container magic has been done by Daniel
> initially - maybe he can help (now on CC:) ...
I honestly never bother with the local QEMU container build sripts as
they've over-engineered IMHO. If I need to debug something locally
I'd just do "podman build -f path/to/docker-file /path/to/docker-dir".
Alternatively just make your changes to the dockerfiles and thne push
the branch to gitlab. Gitlab will run the build and you can pull down
the docker image from your fork's docker registry
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-04 10:00 ` Daniel P. Berrangé
@ 2021-02-04 17:36 ` Richard Henderson
2021-02-04 17:37 ` Daniel P. Berrangé
0 siblings, 1 reply; 10+ messages in thread
From: Richard Henderson @ 2021-02-04 17:36 UTC (permalink / raw)
To: Daniel P. Berrangé, Thomas Huth
Cc: Alex Bennée, Philippe Mathieu-Daudé, qemu-devel
On 2/4/21 12:00 AM, Daniel P. Berrangé wrote:
>>> Hmm. Is there any way to get the full output of the container build? At
>>> present it's being truncated:
>>>
>>> #7 [4/5] RUN yum install -y bzip2 bzip2-devel ccache csnappy-de...
>>>
>>>
>>> In particular, I'm trying to add a new test, and I have added libffi-devel.i686
>>> to the fedora-i386-cross.docker file, but then the actual build fails because
>>> the libffi header file is missing.
...
> Alternatively just make your changes to the dockerfiles and thne push
> the branch to gitlab. Gitlab will run the build and you can pull down
> the docker image from your fork's docker registry
That's what I did, with the results as described above.
r~
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-04 17:36 ` Richard Henderson
@ 2021-02-04 17:37 ` Daniel P. Berrangé
2021-02-04 18:27 ` Richard Henderson
0 siblings, 1 reply; 10+ messages in thread
From: Daniel P. Berrangé @ 2021-02-04 17:37 UTC (permalink / raw)
To: Richard Henderson
Cc: Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé, qemu-devel
On Thu, Feb 04, 2021 at 07:36:19AM -1000, Richard Henderson wrote:
> On 2/4/21 12:00 AM, Daniel P. Berrangé wrote:
> >>> Hmm. Is there any way to get the full output of the container build? At
> >>> present it's being truncated:
> >>>
> >>> #7 [4/5] RUN yum install -y bzip2 bzip2-devel ccache csnappy-de...
> >>>
> >>>
> >>> In particular, I'm trying to add a new test, and I have added libffi-devel.i686
> >>> to the fedora-i386-cross.docker file, but then the actual build fails because
> >>> the libffi header file is missing.
> ...
> > Alternatively just make your changes to the dockerfiles and thne push
> > the branch to gitlab. Gitlab will run the build and you can pull down
> > the docker image from your fork's docker registry
>
> That's what I did, with the results as described above.
Ah, if you can point me to the gitlab pipeline / branch I can have a
look at diagnosing it.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-04 17:37 ` Daniel P. Berrangé
@ 2021-02-04 18:27 ` Richard Henderson
2021-02-04 18:58 ` Daniel P. Berrangé
0 siblings, 1 reply; 10+ messages in thread
From: Richard Henderson @ 2021-02-04 18:27 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé, qemu-devel
On 2/4/21 7:37 AM, Daniel P. Berrangé wrote:
> On Thu, Feb 04, 2021 at 07:36:19AM -1000, Richard Henderson wrote:
>> On 2/4/21 12:00 AM, Daniel P. Berrangé wrote:
>>>>> Hmm. Is there any way to get the full output of the container build? At
>>>>> present it's being truncated:
>>>>>
>>>>> #7 [4/5] RUN yum install -y bzip2 bzip2-devel ccache csnappy-de...
>>>>>
>>>>>
>>>>> In particular, I'm trying to add a new test, and I have added libffi-devel.i686
>>>>> to the fedora-i386-cross.docker file, but then the actual build fails because
>>>>> the libffi header file is missing.
>> ...
>>> Alternatively just make your changes to the dockerfiles and thne push
>>> the branch to gitlab. Gitlab will run the build and you can pull down
>>> the docker image from your fork's docker registry
>>
>> That's what I did, with the results as described above.
>
> Ah, if you can point me to the gitlab pipeline / branch I can have a
> look at diagnosing it.
One that failed is
https://gitlab.com/rth7680/qemu/-/pipelines/250583359
where the cross-i386-tci job fails. It takes some digging to see that all of
the correct bits are in place: from the head (84f9ac62) up to the ffi patch
(7bdae288) which modifies the docker files, up to the gitlab patch (a1d93694)
which adds the cross test. I'll note that for this particular push, gitlab has
failed to rebuild the containers, and I don't know why.
Irritatingly, I re-pushed a combined cross-test+ffi patch on top of my current
tci branch (since I had removed both patches shown above), and it seems to be
working. Possibly because this time gitlab decided to rebuild the containers:
https://gitlab.com/rth7680/qemu/-/pipelines/251596167
r~
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-04 18:27 ` Richard Henderson
@ 2021-02-04 18:58 ` Daniel P. Berrangé
2021-02-04 19:09 ` Richard Henderson
0 siblings, 1 reply; 10+ messages in thread
From: Daniel P. Berrangé @ 2021-02-04 18:58 UTC (permalink / raw)
To: Richard Henderson
Cc: Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé, qemu-devel
On Thu, Feb 04, 2021 at 08:27:00AM -1000, Richard Henderson wrote:
> On 2/4/21 7:37 AM, Daniel P. Berrangé wrote:
> > On Thu, Feb 04, 2021 at 07:36:19AM -1000, Richard Henderson wrote:
> >> On 2/4/21 12:00 AM, Daniel P. Berrangé wrote:
> >>>>> Hmm. Is there any way to get the full output of the container build? At
> >>>>> present it's being truncated:
> >>>>>
> >>>>> #7 [4/5] RUN yum install -y bzip2 bzip2-devel ccache csnappy-de...
> >>>>>
> >>>>>
> >>>>> In particular, I'm trying to add a new test, and I have added libffi-devel.i686
> >>>>> to the fedora-i386-cross.docker file, but then the actual build fails because
> >>>>> the libffi header file is missing.
> >> ...
> >>> Alternatively just make your changes to the dockerfiles and thne push
> >>> the branch to gitlab. Gitlab will run the build and you can pull down
> >>> the docker image from your fork's docker registry
> >>
> >> That's what I did, with the results as described above.
> >
> > Ah, if you can point me to the gitlab pipeline / branch I can have a
> > look at diagnosing it.
>
> One that failed is
>
> https://gitlab.com/rth7680/qemu/-/pipelines/250583359
>
> where the cross-i386-tci job fails. It takes some digging to see that all of
> the correct bits are in place: from the head (84f9ac62) up to the ffi patch
> (7bdae288) which modifies the docker files, up to the gitlab patch (a1d93694)
> which adds the cross test. I'll note that for this particular push, gitlab has
> failed to rebuild the containers, and I don't know why.
> Irritatingly, I re-pushed a combined cross-test+ffi patch on top of my current
> tci branch (since I had removed both patches shown above), and it seems to be
> working. Possibly because this time gitlab decided to rebuild the containers:
>
> https://gitlab.com/rth7680/qemu/-/pipelines/251596167
I'm inclined to blame our gitlab CI rules. We have a condition for
the container builds:
rules:
- changes:
- .gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
- if: '$CI_COMMIT_REF_NAME == "testing/next"'
The intent here is that containers are only built if a patch touches
these files.
The question is though *how* gitlab identifies that changes have been
made, when running a job for a branch. ie what set of commits does it
look at ?
The docs say it looks at the commits "in the push event".
IOW, if you have a git branch which has a delta of 40 commits vs
current git master, and you force push to that branch, it is not
guaranteed to look at all 40 commits. I suspect it only looks
at commits which have changed since the previous push on that
branch.
This is a problem if you are pushing to multiple different
branches.
The containers are always tagged with ":latest", which means
that every time a build runs on a branch it will replace
containers created from a different branch.
So consider you
- Push 40 commits to tci-next with a dockerfile change in the 4th commit.
This triggers a build of the containers.
Containers reflect your tci-next branch content.
- Now push to master to catch up with upstream. This triggers a rebuild
becuase of if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
Containers reflect your master branch content.
- Push 40 commits to tci-next, but only the last 5 in the branch are
different from your previous commit. The container build won't
trigger, despite the branch containing a dockerfile change, because
the commit with the dockerfile wasn't changed from previous push
I've not dug further back into your pipeline history to verify if this
is what happened, but I've convinced myself that it is plausible, and
thus I believe our rules for container rebuilds are likely fundamentally
broken.
When I first introduced the container build stuff to gitlab, I used
the :latest tag because I don't want to pollute the container registry
with an ever growing number of tags for throwaway branches. This was
safe because my original patches *always* ran the container builds,
and told docker to use the existing image as a cache. This meant that
despite running in every pipeline, the container builds were reasonably
fast as they'd hit docker layer cache most of the time.
The addition of the rules to make container builds conditional has
broken the assumption that it is safe to use :latest for the docker
tag.
I think we need to remove the rules condition
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gitlab containers are broken
2021-02-04 18:58 ` Daniel P. Berrangé
@ 2021-02-04 19:09 ` Richard Henderson
0 siblings, 0 replies; 10+ messages in thread
From: Richard Henderson @ 2021-02-04 19:09 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Alex Bennée, Thomas Huth, Philippe Mathieu-Daudé, qemu-devel
On 2/4/21 8:58 AM, Daniel P. Berrangé wrote:
> So consider you
>
> - Push 40 commits to tci-next with a dockerfile change in the 4th commit.
> This triggers a build of the containers.
>
> Containers reflect your tci-next branch content.
>
> - Now push to master to catch up with upstream. This triggers a rebuild
> becuase of if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
>
> Containers reflect your master branch content.
>
> - Push 40 commits to tci-next, but only the last 5 in the branch are
> different from your previous commit. The container build won't
> trigger, despite the branch containing a dockerfile change, because
> the commit with the dockerfile wasn't changed from previous push
>
>
> I've not dug further back into your pipeline history to verify if this
> is what happened, but I've convinced myself that it is plausible
This is exactly my development process, switching between tasks on different
branches as I wait for build results.
> The addition of the rules to make container builds conditional has
> broken the assumption that it is safe to use :latest for the docker
> tag.
>
> I think we need to remove the rules condition
Thanks for the analysis.
r~
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-02-04 19:38 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-03 23:04 gitlab containers are broken Richard Henderson
2021-02-04 6:03 ` Thomas Huth
2021-02-04 6:27 ` Richard Henderson
2021-02-04 8:08 ` Thomas Huth
2021-02-04 10:00 ` Daniel P. Berrangé
2021-02-04 17:36 ` Richard Henderson
2021-02-04 17:37 ` Daniel P. Berrangé
2021-02-04 18:27 ` Richard Henderson
2021-02-04 18:58 ` Daniel P. Berrangé
2021-02-04 19:09 ` Richard Henderson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).