* changing tests/qtest/meson.build causes unnecessary rebuilding
@ 2021-01-21 11:56 Peter Maydell
2021-01-21 13:12 ` Paolo Bonzini
0 siblings, 1 reply; 5+ messages in thread
From: Peter Maydell @ 2021-01-21 11:56 UTC (permalink / raw)
To: QEMU Developers; +Cc: Paolo Bonzini
$ make -C build/arm-clang/ -j8
make: Entering directory
'/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
[1/23] Generating qemu-version.h with a meson_exe.py custom command
make: Leaving directory
'/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
$ touch tests/qtest/meson.build
$ make -C build/arm-clang/ -j8
make: Entering directory
'/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
/usr/bin/ninja build.ninja && touch build.ninja.stamp
[0/1] Regenerating build files.
The Meson build system
[...]
It then goes on to rebuild hundreds or thousands of files, most of
which are not even in tests/. (Oddly, the exact set of files recompiled
seems to vary from run to run.)
Forcing a full rebuild of all of QEMU seems rather excessive when
the change was likely "add a new test case"...
thanks
-- PMM
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: changing tests/qtest/meson.build causes unnecessary rebuilding
2021-01-21 11:56 changing tests/qtest/meson.build causes unnecessary rebuilding Peter Maydell
@ 2021-01-21 13:12 ` Paolo Bonzini
2021-01-21 14:29 ` Thomas Huth
2021-01-21 15:55 ` Philippe Mathieu-Daudé
0 siblings, 2 replies; 5+ messages in thread
From: Paolo Bonzini @ 2021-01-21 13:12 UTC (permalink / raw)
To: Peter Maydell, QEMU Developers
On 21/01/21 12:56, Peter Maydell wrote:
> $ make -C build/arm-clang/ -j8
> make: Entering directory
> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
> [1/23] Generating qemu-version.h with a meson_exe.py custom command
> make: Leaving directory
> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
> $ touch tests/qtest/meson.build
> $ make -C build/arm-clang/ -j8
> make: Entering directory
> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
> /usr/bin/ninja build.ninja && touch build.ninja.stamp
> [0/1] Regenerating build files.
> The Meson build system
> [...]
>
> It then goes on to rebuild hundreds or thousands of files, most of
> which are not even in tests/. (Oddly, the exact set of files recompiled
> seems to vary from run to run.)
>
> Forcing a full rebuild of all of QEMU seems rather excessive when
> the change was likely "add a new test case"...
This is "avoid build.ninja changes due to order of hash table iteration"
(https://github.com/mesonbuild/meson/pull/7900/). I think Meson 0.57
(with the fix) should be out soon, hopefully before 6.0.
Alternatively you can try to bug your distro to include the patches,
they are pretty safe.
Paolo
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: changing tests/qtest/meson.build causes unnecessary rebuilding
2021-01-21 13:12 ` Paolo Bonzini
@ 2021-01-21 14:29 ` Thomas Huth
2021-01-21 15:31 ` Paolo Bonzini
2021-01-21 15:55 ` Philippe Mathieu-Daudé
1 sibling, 1 reply; 5+ messages in thread
From: Thomas Huth @ 2021-01-21 14:29 UTC (permalink / raw)
To: Paolo Bonzini, Peter Maydell, QEMU Developers
Cc: Alex Bennée, Philippe Mathieu-Daudé
On 21/01/2021 14.12, Paolo Bonzini wrote:
> On 21/01/21 12:56, Peter Maydell wrote:
>> $ make -C build/arm-clang/ -j8
>> make: Entering directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> [1/23] Generating qemu-version.h with a meson_exe.py custom command
>> make: Leaving directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> $ touch tests/qtest/meson.build
>> $ make -C build/arm-clang/ -j8
>> make: Entering directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> /usr/bin/ninja build.ninja && touch build.ninja.stamp
>> [0/1] Regenerating build files.
>> The Meson build system
>> [...]
>>
>> It then goes on to rebuild hundreds or thousands of files, most of
>> which are not even in tests/. (Oddly, the exact set of files recompiled
>> seems to vary from run to run.)
>>
>> Forcing a full rebuild of all of QEMU seems rather excessive when
>> the change was likely "add a new test case"...
>
> This is "avoid build.ninja changes due to order of hash table iteration"
> (https://github.com/mesonbuild/meson/pull/7900/). I think Meson 0.57 (with
> the fix) should be out soon, hopefully before 6.0.
>
> Alternatively you can try to bug your distro to include the patches, they
> are pretty safe.
Not sure if it is related, but I noticed that we are also rebuilding a lot
of files in the gitlab-CI that we did not before the meson conversion,
especially in the check-system-* jobs, e.g:
https://gitlab.com/qemu-project/qemu/-/jobs/977344949#L366
The check-system-* jobs should normally take the artifacts from the
build-system-* jobs and thus hardly recompile anything at all.
A part of the problem seems to be that we check out the submodules again, I
can get rid of the superfluous reconfiguration step by adding something like:
diff -u a/.gitlab-ci.yml b/.gitlab-ci.yml
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -39,6 +39,8 @@ include:
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
script:
- cd build
+ - touch *
+ - make git-submodule-update
- find . -type f -exec touch {} +
- make $MAKE_CHECK_ARGS
... but still, the jobs then recompile almost all files afterwards... could
that be related to that meson problem, too?
Thomas
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: changing tests/qtest/meson.build causes unnecessary rebuilding
2021-01-21 14:29 ` Thomas Huth
@ 2021-01-21 15:31 ` Paolo Bonzini
0 siblings, 0 replies; 5+ messages in thread
From: Paolo Bonzini @ 2021-01-21 15:31 UTC (permalink / raw)
To: Thomas Huth, Peter Maydell, QEMU Developers
Cc: Alex Bennée, Philippe Mathieu-Daudé
On 21/01/21 15:29, Thomas Huth wrote:
>
> Not sure if it is related, but I noticed that we are also rebuilding a
> lot of files in the gitlab-CI that we did not before the meson
> conversion, especially in the check-system-* jobs, e.g:
>
> https://gitlab.com/qemu-project/qemu/-/jobs/977344949#L366
>
> The check-system-* jobs should normally take the artifacts from the
> build-system-* jobs and thus hardly recompile anything at all.
>
> A part of the problem seems to be that we check out the submodules
> again, I can get rid of the superfluous reconfiguration step by adding
> something like:
Yes, there's a
[0/1] Regenerating build files.
/usr/bin/python3: can't open file
'/builds/qemu-project/qemu/meson/meson.py': [Errno 2] No such file or
directory
because the submodules have not been checked out, and that causes meson
to run again.
> diff -u a/.gitlab-ci.yml b/.gitlab-ci.yml
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -39,6 +39,8 @@ include:
> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> script:
> - cd build
> + - touch *
> + - make git-submodule-update
> - find . -type f -exec touch {} +
> - make $MAKE_CHECK_ARGS
>
> ... but still, the jobs then recompile almost all files afterwards...
> could that be related to that meson problem, too?
No, I think it's just that, after the new checkout, the source files'
timestamps should be quite new and cause everything to be rebuilt.
Paolo
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: changing tests/qtest/meson.build causes unnecessary rebuilding
2021-01-21 13:12 ` Paolo Bonzini
2021-01-21 14:29 ` Thomas Huth
@ 2021-01-21 15:55 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 5+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-01-21 15:55 UTC (permalink / raw)
To: Paolo Bonzini, Peter Maydell, QEMU Developers
Hi Peter,
On 1/21/21 2:12 PM, Paolo Bonzini wrote:
> On 21/01/21 12:56, Peter Maydell wrote:
>> $ make -C build/arm-clang/ -j8
>> make: Entering directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> [1/23] Generating qemu-version.h with a meson_exe.py custom command
>> make: Leaving directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> $ touch tests/qtest/meson.build
>> $ make -C build/arm-clang/ -j8
>> make: Entering directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> /usr/bin/ninja build.ninja && touch build.ninja.stamp
>> [0/1] Regenerating build files.
>> The Meson build system
>> [...]
>>
>> It then goes on to rebuild hundreds or thousands of files, most of
>> which are not even in tests/. (Oddly, the exact set of files recompiled
>> seems to vary from run to run.)
>>
>> Forcing a full rebuild of all of QEMU seems rather excessive when
>> the change was likely "add a new test case"...
>
> This is "avoid build.ninja changes due to order of hash table iteration"
> (https://github.com/mesonbuild/meson/pull/7900/). I think Meson 0.57
> (with the fix) should be out soon, hopefully before 6.0.
>
> Alternatively you can try to bug your distro to include the patches,
> they are pretty safe.
Possible kludge rebuilding meson locally:
Paolo's #7900 got merged in commit 1c582a9de:
("Merge pull request #7900 from bonzini/stabilize-hash"),
so a possible kludge is to build and install meson in your
$HOME/.local/bin/ directory with pip as:
$ python3 -m pip install --user \
git+https://github.com/mesonbuild/meson.git@1c582a9de
Then either update your $PATH or use:
$ ./configure --meson=$HOME/.local/bin/meson ...
Regards,
Phil.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-01-21 15:56 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-21 11:56 changing tests/qtest/meson.build causes unnecessary rebuilding Peter Maydell
2021-01-21 13:12 ` Paolo Bonzini
2021-01-21 14:29 ` Thomas Huth
2021-01-21 15:31 ` Paolo Bonzini
2021-01-21 15:55 ` Philippe Mathieu-Daudé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.