check-tcg errors (build-user, build-user-plugins) again

check-tcg errors (build-user, build-user-plugins) again

[Claudio Fontana]

Hi Alex and all,

when trying to use check-tcg (master), I am getting often these errors:

$ ../configure --disable-system --disable-tools

$ make -j12 check-tcg

ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian11...
Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross...
Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian10...
ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 

[...]
  TEST    linux-test on x86_64
timeout: failed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’timeout: : No such file or directoryfailed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’

[...]


Is there some pre-configuration on the host necessary to be able to run check-tcg?

I see these errors in gitlab also for

build-user
build-user-plugin

Maybe this is what Philippe mentioned before though, that this is expected at the moment due to a temporary Meson shortcoming?

Ciao,

Claudio


-- 
Claudio Fontana
Engineering Manager Virtualization, SUSE Labs Core

SUSE Software Solutions Italy Srl

Re: check-tcg errors (build-user, build-user-plugins) again

[Alex Bennée]

Claudio Fontana <cfontana@suse.de> writes:

> Hi Alex and all,
>
> when trying to use check-tcg (master), I am getting often these errors:
>
> $ ../configure --disable-system --disable-tools
>
> $ make -j12 check-tcg
>
> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian11...
> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross...
> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian10...
> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>
> [...]
>   TEST    linux-test on x86_64
> timeout: failed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’timeout: : No such file or directoryfailed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’
>
> [...]
>
>
> Is there some pre-configuration on the host necessary to be able to
> run check-tcg?

There shouldn't be but those errors remind me of some of the tweaks I
had to make to me Gentoo system when using podman (instead of docker).
In the end I think I just ended up adding the lines:
  
  alex:100000:65536

to /etc/subgid and /etc/subgid-

Marc-André may have some better pointers as he added podman support to
the builder scripts.

The main difference between the images on the registry and the local
versions is most add the current user so there is a clean mapping
between the container user and the host file-system. It's the last step
of the build so we still use the cached layers from the registry
versions.

> I see these errors in gitlab also for
>
> build-user
> build-user-plugin
>
> Maybe this is what Philippe mentioned before though, that this is
> expected at the moment due to a temporary Meson shortcoming?

That is odd - I'm not seeing anything like that on the master builds:

  https://gitlab.com/qemu-project/qemu/-/jobs/883985106
  https://gitlab.com/qemu-project/qemu/-/jobs/883985113

AFAIK GitLab is still using Docker to build it's containers (albeit with
BUILDKIT enabled).
  
>
> Ciao,
>
> Claudio


-- 
Alex Bennée

Re: check-tcg errors (build-user, build-user-plugins) again

[Claudio Fontana]

On 12/2/20 12:16 PM, Alex Bennée wrote:
> 
> Claudio Fontana <cfontana@suse.de> writes:
> 
>> Hi Alex and all,
>>
>> when trying to use check-tcg (master), I am getting often these errors:
>>
>> $ ../configure --disable-system --disable-tools
>>
>> $ make -j12 check-tcg
>>
>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian11...
>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross...
>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian10...
>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>
>> [...]
>>   TEST    linux-test on x86_64
>> timeout: failed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’timeout: : No such file or directoryfailed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’
>>
>> [...]
>>
>>
>> Is there some pre-configuration on the host necessary to be able to
>> run check-tcg?
> 
> There shouldn't be but those errors remind me of some of the tweaks I
> had to make to me Gentoo system when using podman (instead of docker).
> In the end I think I just ended up adding the lines:
>   
>   alex:100000:65536
> 
> to /etc/subgid and /etc/subgid-
> 
> Marc-André may have some better pointers as he added podman support to
> the builder scripts.


I did that and things seem a bit better, but still a lot of errors:


63      ../sysdeps/x86_64/start.S: No such file or directory.

Error: error creating build container: The following failures happened while trying to pull image specified by "debian:bullseye-slim" based on search registries in /etc/containers/registries.conf:
* "localhost/debian:bullseye-slim": Error initializing source docker://localhost/debian:bullseye-slim: error pinging docker registry localhost: Get https://localhost/v2/: dial tcp [::1]:443: connect: connection refused
* "docker.io/library/debian:bullseye-slim": Error committing the finished image: error adding layer with blob "sha256:ae63fcbbc3b289e425e4c8840ccde4314f4a060cbc0345e6871a28bdc72f6fe8": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
Traceback (most recent call last):
  File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 709, in <module>
    sys.exit(main())
  File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 705, in main
    return args.cmdobj.run(args, argv)
  File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 501, in run
    extra_files_cksum=cksum)
  File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 354, in build_image
    quiet=quiet)
  File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 244, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)


[...]
Error: error pulling image "registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross": unable to pull registry.gitlab.com/qemu-project/

[...]



> 
> The main difference between the images on the registry and the local
> versions is most add the current user so there is a clean mapping
> between the container user and the host file-system. It's the last step
> of the build so we still use the cached layers from the registry
> versions.
> 
>> I see these errors in gitlab also for
>>
>> build-user
>> build-user-plugin
>>
>> Maybe this is what Philippe mentioned before though, that this is
>> expected at the moment due to a temporary Meson shortcoming?
> 
> That is odd - I'm not seeing anything like that on the master builds:
> 
>   https://gitlab.com/qemu-project/qemu/-/jobs/883985106
>   https://gitlab.com/qemu-project/qemu/-/jobs/883985113
> 
> AFAIK GitLab is still using Docker to build it's containers (albeit with
> BUILDKIT enabled).


I am running again on gitlab the master branch, maybe there is something I need to fix, but to do that I need to enable check-tcg successfully I think.

Thanks!

Claudio

>   
>>
>> Ciao,
>>
>> Claudio
> 
> 

Re: check-tcg errors (build-user, build-user-plugins) again

[Alex Bennée]

Claudio Fontana <cfontana@suse.de> writes:

> On 12/2/20 12:16 PM, Alex Bennée wrote:
>> 
>> Claudio Fontana <cfontana@suse.de> writes:
>> 
>>> Hi Alex and all,
>>>
>>> when trying to use check-tcg (master), I am getting often these errors:
>>>
>>> $ ../configure --disable-system --disable-tools
>>>
>>> $ make -j12 check-tcg
>>>
>>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian11...
>>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross...
>>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian10...
>>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>>
>>> [...]
>>>   TEST    linux-test on x86_64
>>> timeout: failed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’timeout: : No such file or directoryfailed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’
>>>
>>> [...]
>>>
>>>
>>> Is there some pre-configuration on the host necessary to be able to
>>> run check-tcg?
>> 
>> There shouldn't be but those errors remind me of some of the tweaks I
>> had to make to me Gentoo system when using podman (instead of docker).
>> In the end I think I just ended up adding the lines:
>>   
>>   alex:100000:65536
>> 
>> to /etc/subgid and /etc/subgid-
>> 
>> Marc-André may have some better pointers as he added podman support to
>> the builder scripts.
>
>
> I did that and things seem a bit better, but still a lot of errors:
>
>
> 63      ../sysdeps/x86_64/start.S: No such file or directory.
>
> Error: error creating build container: The following failures happened while trying to pull image specified by "debian:bullseye-slim" based on search registries in /etc/containers/registries.conf:
> * "localhost/debian:bullseye-slim": Error initializing source docker://localhost/debian:bullseye-slim: error pinging docker registry localhost: Get https://localhost/v2/: dial tcp [::1]:443: connect: connection refused
> * "docker.io/library/debian:bullseye-slim": Error committing the finished image: error adding layer with blob "sha256:ae63fcbbc3b289e425e4c8840ccde4314f4a060cbc0345e6871a28bdc72f6fe8": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
> Traceback (most recent call last):
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 709, in <module>
>     sys.exit(main())
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 705, in main
>     return args.cmdobj.run(args, argv)
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 501, in run
>     extra_files_cksum=cksum)
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 354, in build_image
>     quiet=quiet)
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 244, in _do_check
>     return subprocess.check_call(self._command + cmd, **kwargs)
>   File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
>     raise CalledProcessError(retcode, cmd)
>
>
> [...]
> Error: error pulling image "registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross": unable to pull registry.gitlab.com/qemu-project/
>

I'm guessing this can be fixed by adding gitlab to /etc/containers/registries.conf

I'll see if I can resurrect my podman setup because it was working before
we added the caching from gitlab.

> [...]
>
>
>
>> 
>> The main difference between the images on the registry and the local
>> versions is most add the current user so there is a clean mapping
>> between the container user and the host file-system. It's the last step
>> of the build so we still use the cached layers from the registry
>> versions.
>> 
>>> I see these errors in gitlab also for
>>>
>>> build-user
>>> build-user-plugin
>>>
>>> Maybe this is what Philippe mentioned before though, that this is
>>> expected at the moment due to a temporary Meson shortcoming?
>> 
>> That is odd - I'm not seeing anything like that on the master builds:
>> 
>>   https://gitlab.com/qemu-project/qemu/-/jobs/883985106
>>   https://gitlab.com/qemu-project/qemu/-/jobs/883985113
>> 
>> AFAIK GitLab is still using Docker to build it's containers (albeit with
>> BUILDKIT enabled).
>
>
> I am running again on gitlab the master branch, maybe there is something I need to fix, but to do that I need to enable check-tcg successfully I think.
>
> Thanks!
>
> Claudio
>
>>   
>>>
>>> Ciao,
>>>
>>> Claudio
>> 
>> 


-- 
Alex Bennée

Re: check-tcg errors (build-user, build-user-plugins) again

[Claudio Fontana]

On 12/2/20 1:52 PM, Alex Bennée wrote:
> 
> Claudio Fontana <cfontana@suse.de> writes:
> 
>> On 12/2/20 12:16 PM, Alex Bennée wrote:
>>>
>>> Claudio Fontana <cfontana@suse.de> writes:
>>>
>>>> Hi Alex and all,
>>>>
>>>> when trying to use check-tcg (master), I am getting often these errors:
>>>>
>>>> $ ../configure --disable-system --disable-tools
>>>>
>>>> $ make -j12 check-tcg
>>>>
>>>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian11...
>>>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross...
>>>> Trying to pull registry.gitlab.com/qemu-project/qemu/qemu/debian10...
>>>> ERRO[0000] cannot find mappings for user claudio: No subgid ranges found for group "claudio" in /etc/subgid 
>>>>
>>>> [...]
>>>>   TEST    linux-test on x86_64
>>>> timeout: failed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’timeout: : No such file or directoryfailed to run command ‘/home/claudio/git/qemu/build/qemu-x86_64’
>>>>
>>>> [...]
>>>>
>>>>
>>>> Is there some pre-configuration on the host necessary to be able to
>>>> run check-tcg?
>>>
>>> There shouldn't be but those errors remind me of some of the tweaks I
>>> had to make to me Gentoo system when using podman (instead of docker).
>>> In the end I think I just ended up adding the lines:
>>>   
>>>   alex:100000:65536
>>>
>>> to /etc/subgid and /etc/subgid-
>>>
>>> Marc-André may have some better pointers as he added podman support to
>>> the builder scripts.
>>
>>
>> I did that and things seem a bit better, but still a lot of errors:
>>
>>
>> 63      ../sysdeps/x86_64/start.S: No such file or directory.
>>
>> Error: error creating build container: The following failures happened while trying to pull image specified by "debian:bullseye-slim" based on search registries in /etc/containers/registries.conf:
>> * "localhost/debian:bullseye-slim": Error initializing source docker://localhost/debian:bullseye-slim: error pinging docker registry localhost: Get https://localhost/v2/: dial tcp [::1]:443: connect: connection refused
>> * "docker.io/library/debian:bullseye-slim": Error committing the finished image: error adding layer with blob "sha256:ae63fcbbc3b289e425e4c8840ccde4314f4a060cbc0345e6871a28bdc72f6fe8": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
>> Traceback (most recent call last):
>>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 709, in <module>
>>     sys.exit(main())
>>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 705, in main
>>     return args.cmdobj.run(args, argv)
>>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 501, in run
>>     extra_files_cksum=cksum)
>>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 354, in build_image
>>     quiet=quiet)
>>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 244, in _do_check
>>     return subprocess.check_call(self._command + cmd, **kwargs)
>>   File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
>>     raise CalledProcessError(retcode, cmd)
>>
>>
>> [...]
>> Error: error pulling image "registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross": unable to pull registry.gitlab.com/qemu-project/
>>
> 
> I'm guessing this can be fixed by adding gitlab to /etc/containers/registries.conf

Hi Alex, I added:

[registries.search]
registries = ["docker.io", "registry.gitlab.com"]

I get:

Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
Error: error pulling image "registry.gitlab.com/qemu-project/qemu/qemu/debian11": unable to pull registry.gitlab.com/qemu-project/qemu/qemu/debian11: unable to pull image: Error committing the finished image: error adding layer with blob "sha256:ae63fcbbc3b289e425e4c8840ccde4314f4a060cbc0345e6871a28bdc72f6fe8": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument

Any idea?




> 
> I'll see if I can resurrect my podman setup because it was working before
> we added the caching from gitlab.
> 
>> [...]
>>
>>
>>
>>>
>>> The main difference between the images on the registry and the local
>>> versions is most add the current user so there is a clean mapping
>>> between the container user and the host file-system. It's the last step
>>> of the build so we still use the cached layers from the registry
>>> versions.
>>>
>>>> I see these errors in gitlab also for
>>>>
>>>> build-user
>>>> build-user-plugin
>>>>
>>>> Maybe this is what Philippe mentioned before though, that this is
>>>> expected at the moment due to a temporary Meson shortcoming?
>>>
>>> That is odd - I'm not seeing anything like that on the master builds:
>>>
>>>   https://gitlab.com/qemu-project/qemu/-/jobs/883985106
>>>   https://gitlab.com/qemu-project/qemu/-/jobs/883985113
>>>
>>> AFAIK GitLab is still using Docker to build it's containers (albeit with
>>> BUILDKIT enabled).
>>
>>
>> I am running again on gitlab the master branch, maybe there is something I need to fix, but to do that I need to enable check-tcg successfully I think.
>>
>> Thanks!
>>
>> Claudio
>>
>>>   
>>>>
>>>> Ciao,
>>>>
>>>> Claudio
>>>
>>>
> 
> 

Re: check-tcg errors (build-user, build-user-plugins) again

[Philippe Mathieu-Daudé]

On 12/2/20 12:25 PM, Claudio Fontana wrote:
> On 12/2/20 12:16 PM, Alex Bennée wrote:
>> Claudio Fontana <cfontana@suse.de> writes:
>>> Is there some pre-configuration on the host necessary to be able to
>>> run check-tcg?
>>
>> There shouldn't be but those errors remind me of some of the tweaks I
>> had to make to me Gentoo system when using podman (instead of docker).
>> In the end I think I just ended up adding the lines:
>>   
>>   alex:100000:65536
>>
>> to /etc/subgid and /etc/subgid-
>>
>> Marc-André may have some better pointers as he added podman support to
>> the builder scripts.

Not sure if helpful, but what worked for me is remove podman
and use docker... For sure I'm not testing podman, enough to
deal with docker.

> I did that and things seem a bit better, but still a lot of errors:
> 
> 
> 63      ../sysdeps/x86_64/start.S: No such file or directory.
> 
> Error: error creating build container: The following failures happened while trying to pull image specified by "debian:bullseye-slim" based on search registries in /etc/containers/registries.conf:
> * "localhost/debian:bullseye-slim": Error initializing source docker://localhost/debian:bullseye-slim: error pinging docker registry localhost: Get https://localhost/v2/: dial tcp [::1]:443: connect: connection refused
> * "docker.io/library/debian:bullseye-slim": Error committing the finished image: error adding layer with blob "sha256:ae63fcbbc3b289e425e4c8840ccde4314f4a060cbc0345e6871a28bdc72f6fe8": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
> Traceback (most recent call last):
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 709, in <module>
>     sys.exit(main())
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 705, in main
>     return args.cmdobj.run(args, argv)
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 501, in run
>     extra_files_cksum=cksum)
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 354, in build_image
>     quiet=quiet)
>   File "/home/claudio/git/qemu-pristine/qemu/tests/docker/docker.py", line 244, in _do_check
>     return subprocess.check_call(self._command + cmd, **kwargs)
>   File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
>     raise CalledProcessError(retcode, cmd)
> 
> 
> [...]
> Error: error pulling image "registry.gitlab.com/qemu-project/qemu/qemu/fedora-cris-cross": unable to pull registry.gitlab.com/qemu-project/

Maybe you need "use explicit docker.io registry" from Daniel:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg763484.html

[...]

check-tcg HOWTO?

[Claudio Fontana]

Hi Alex,

happy new year,

I am trying to get check-tcg to run reliably,
as I am doing some substantial refactoring of tcg cpu operations, so I need to verify that TCG is fine.

This is an overall getting started question, is there a how-to on how to use check-tcg and how to fix things when things don't go smoothly?

I get different results on different machines for check-tcg, although the runs are containerized,
on one machine the tests for aarch64 tcg are SKIPPED completely (so no errors),

on the other machine I get:

qemu-system-aarch64: terminating on signal 15 from pid 18583 (timeout)
qemu-system-aarch64: terminating on signal 15 from pid 18584 (timeout)
qemu-system-aarch64: terminating on signal 15 from pid 18585 (timeout)
make[2]: *** [../Makefile.target:162: run-hello] Error 124
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [../Makefile.target:162: run-pauth-3] Error 124
make[2]: *** [../Makefile.target:162: run-memory] Error 124

Both are configured with 

configure --enable-tcg

Anything more than V=1 to get more output?
How do I debug and get logs and cores out of containers?

in tests/tcg/ there is:

a README (with no hint unfortunately) ,
Makefile.qemu
Makefile.prereqs
Makefile.target

There are a bunch of variables in these files, which seem to be possible to configure, am I expected to set some of those?

I think that it would be beneficial to have either more documentation or more immediately actionable information out of make check failures;

Any help you could give me to make some progess?

Thanks,

Claudio

Re: check-tcg HOWTO?

[Alex Bennée]

Claudio Fontana <cfontana@suse.de> writes:

> Hi Alex,
>
> happy new year,
>
> I am trying to get check-tcg to run reliably,
> as I am doing some substantial refactoring of tcg cpu operations, so I need to verify that TCG is fine.
>
> This is an overall getting started question, is there a how-to on how
> to use check-tcg and how to fix things when things don't go smoothly?

Not really but I could certainly add something. Generally I just run the
tests manually in gdb when I'm trying to debug stuff.

> I get different results on different machines for check-tcg, although the runs are containerized,
> on one machine the tests for aarch64 tcg are SKIPPED completely (so no
> errors),

The compiles *may* be containerized - the runs are always in your host
environment. It's one of the reasons the binaries are built as static
images so you don't need to mess about with dynamic linking and
libraries.

The only reason some tests get skipped is if you have a locally
installed cross compiler which doesn't support some architecture
features (e.g. CROSS_CC_HAS_SVE).

> on the other machine I get:
>
> qemu-system-aarch64: terminating on signal 15 from pid 18583 (timeout)
> qemu-system-aarch64: terminating on signal 15 from pid 18584 (timeout)
> qemu-system-aarch64: terminating on signal 15 from pid 18585 (timeout)
> make[2]: *** [../Makefile.target:162: run-hello] Error 124
> make[2]: *** Waiting for unfinished jobs....
> make[2]: *** [../Makefile.target:162: run-pauth-3] Error 124
> make[2]: *** [../Makefile.target:162: run-memory] Error 124

Given it's timing out on hello I guess it's the shutdown deadlocking.
Running with V=1 will give you the command line but the semihosting
config is setup for redirect. So I usually build my own command line. e.g.:

  ./qemu-system-aarch64 -monitor none -display none \
    -chardev stdio,id=output  \
    -M virt -cpu max -display none \
    -semihosting-config enable=on,target=native,chardev=output \
    -kernel tests/tcg/aarch64-softmmu/hello

There is nothing particularly special apart from making sure semihosting
is wired up for the output. Apart from some special cases like
test-mmap-XXXX most tests don't take any arguments.

>
> Both are configured with 
>
> configure --enable-tcg
>
> Anything more than V=1 to get more output?

The output is normally dumped in $TESTNAME.out in the appropriate
$BUILDDIR/tests/tcg/$TARGET/ directory.

> How do I debug and get logs and cores out of containers?

As I mentioned above the tests are not run in containers, just the
compiles (if local compilers are missing).

>
> in tests/tcg/ there is:
>
> a README (with no hint unfortunately) ,

Woefully out of date I'm afraid. What docs we have are in docs/devel/testing.rst

> Makefile.qemu

Links into the main tests/Makefile.include - invoked for each target

> Makefile.prereqs

This ensures docker images are built (if required) for each set of tests.

> Makefile.target

This is the main (rather simple) makefile which provides the build and
run targets. You can run directly if you are in the right build dir, eg:

  13:58:10 [alex@zen:~/l/q/b/a/t/t/aarch64-softmmu] |✔ + pwd
  /home/alex/lsrc/qemu.git/builds/arm.all/tests/tcg/aarch64-softmmu
  13:58:57 [alex@zen:~/l/q/b/a/t/t/aarch64-softmmu] |✔ +
  make  -f ~/lsrc/qemu.git/tests/tcg/Makefile.target TARGET="aarch64-softmmu" SRC_PATH="/home/alex/lsrc/qemu.git" run

But TBH this is functionally equivalent to calling:

  make run-tcg-tests-aarch64-softmmu

in your main build directory.

> There are a bunch of variables in these files, which seem to be
> possible to configure, am I expected to set some of those?

Not really. Most of the magic variables from:

  tests/tcg/config-$TARGET.mak

which is built by tests/tcg/configure.sh during the configure step.

> I think that it would be beneficial to have either more documentation
> or more immediately actionable information out of make check failures;

V=1 should show the command lines run and then you should be able to run
the tests directly yourself.

> Any help you could give me to make some progess?

Hope that helps.

>
> Thanks,
>
> Claudio


-- 
Alex Bennée

Re: check-tcg HOWTO?

[Claudio Fontana]

Ciao Alex,

thanks for your answer,

On 1/11/21 2:35 PM, Alex Bennée wrote:
> 
> Claudio Fontana <cfontana@suse.de> writes:
> 
>> Hi Alex,
>>
>> happy new year,
>>
>> I am trying to get check-tcg to run reliably,
>> as I am doing some substantial refactoring of tcg cpu operations, so I need to verify that TCG is fine.
>>
>> This is an overall getting started question, is there a how-to on how
>> to use check-tcg and how to fix things when things don't go smoothly?
> 
> Not really but I could certainly add something. Generally I just run the
> tests manually in gdb when I'm trying to debug stuff.

Right, I plan to do the same, if I get to the command line to run.
I think it does make sense to add something similar to what was explained here in documentation, with a pointer maybe from README?


> 
>> I get different results on different machines for check-tcg, although the runs are containerized,
>> on one machine the tests for aarch64 tcg are SKIPPED completely (so no
>> errors),
> 
> The compiles *may* be containerized - the runs are always in your host
> environment. It's one of the reasons the binaries are built as static
> images so you don't need to mess about with dynamic linking and
> libraries.
> 

Ah good to know, thanks. So everything is actually run in the host environment in the end.

> The only reason some tests get skipped is if you have a locally
> installed cross compiler which doesn't support some architecture
> features (e.g. CROSS_CC_HAS_SVE).


hmm I will have to check then how to make sure that the test does not see these cross compilers...?


> 
>> on the other machine I get:
>>
>> qemu-system-aarch64: terminating on signal 15 from pid 18583 (timeout)
>> qemu-system-aarch64: terminating on signal 15 from pid 18584 (timeout)
>> qemu-system-aarch64: terminating on signal 15 from pid 18585 (timeout)
>> make[2]: *** [../Makefile.target:162: run-hello] Error 124
>> make[2]: *** Waiting for unfinished jobs....
>> make[2]: *** [../Makefile.target:162: run-pauth-3] Error 124
>> make[2]: *** [../Makefile.target:162: run-memory] Error 124
> 
> Given it's timing out on hello I guess it's the shutdown deadlocking.
> Running with V=1 will give you the command line but the semihosting
> config is setup for redirect. So I usually build my own command line. e.g.:
> 
>   ./qemu-system-aarch64 -monitor none -display none \
>     -chardev stdio,id=output  \
>     -M virt -cpu max -display none \
>     -semihosting-config enable=on,target=native,chardev=output \
>     -kernel tests/tcg/aarch64-softmmu/hello
> 


Would it be possible for check-tcg (and possibly even make check in general where applicable)
to automatically spew the command line to reproduce the error, similar to what you have shown here?


I think this is would be of great value for anyone to be able to act on the errors reported.


> There is nothing particularly special apart from making sure semihosting
> is wired up for the output. Apart from some special cases like
> test-mmap-XXXX most tests don't take any arguments.
> 
>>
>> Both are configured with 
>>
>> configure --enable-tcg
>>
>> Anything more than V=1 to get more output?
> 
> The output is normally dumped in $TESTNAME.out in the appropriate
> $BUILDDIR/tests/tcg/$TARGET/ directory.
> 
>> How do I debug and get logs and cores out of containers?
> 
> As I mentioned above the tests are not run in containers, just the
> compiles (if local compilers are missing).

Thanks for clearing this up!

> 
>>
>> in tests/tcg/ there is:
>>
>> a README (with no hint unfortunately) ,
> 
> Woefully out of date I'm afraid. What docs we have are in docs/devel/testing.rst

maybe a

+ Please see docs/devel/testing.rst for hints on how to run make-tcg and reproduce its results from the cmdline.

> 
>> Makefile.qemu
> 
> Links into the main tests/Makefile.include - invoked for each target
> 
>> Makefile.prereqs
> 
> This ensures docker images are built (if required) for each set of tests.
> 
>> Makefile.target
> 
> This is the main (rather simple) makefile which provides the build and
> run targets. You can run directly if you are in the right build dir, eg:
> 
>   13:58:10 [alex@zen:~/l/q/b/a/t/t/aarch64-softmmu] |✔ + pwd
>   /home/alex/lsrc/qemu.git/builds/arm.all/tests/tcg/aarch64-softmmu
>   13:58:57 [alex@zen:~/l/q/b/a/t/t/aarch64-softmmu] |✔ +
>   make  -f ~/lsrc/qemu.git/tests/tcg/Makefile.target TARGET="aarch64-softmmu" SRC_PATH="/home/alex/lsrc/qemu.git" run
> 
> But TBH this is functionally equivalent to calling:
> 
>   make run-tcg-tests-aarch64-softmmu
> 
> in your main build directory.

Thanks, that's helpful.

> 
>> There are a bunch of variables in these files, which seem to be
>> possible to configure, am I expected to set some of those?
> 
> Not really. Most of the magic variables from:
> 
>   tests/tcg/config-$TARGET.mak
> 
> which is built by tests/tcg/configure.sh during the configure step.
> 
>> I think that it would be beneficial to have either more documentation
>> or more immediately actionable information out of make check failures;
> 
> V=1 should show the command lines run and then you should be able to run
> the tests directly yourself.


Hmm V=1 did not show the command line to me, but maybe I just missed it somehow?
Or it contained some sockets that cannot be manually connected?

> 
>> Any help you could give me to make some progess?
> 
> Hope that helps.

It does, I wonder if this could be fed into docs/ with a pointer to it from README,
and also if this new feature for the tests could be developed, ie, producing a command line useful for reproducing the error,
something that can be run directly without further editing, sanitizing etc...

Thanks a lot,

Claudio