Re: [Qemu-devel] [PATCH v2 0/2] Acceptance Tests: Test the Raspberry Pi 2 board

Re: [Qemu-devel] [PATCH v2 0/2] Acceptance Tests: Test the Raspberry Pi 2 board

[Philippe Mathieu-Daudé]

On 3/13/19 12:45 AM, Philippe Mathieu-Daudé wrote:
> Add the raspi2 to the list of boards tested by the
> boot_linux_console Avocado test.
> 
> Based on "Acceptance Tests: target architecture support" v5:
> https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04111.html
> Based-on: <20190312171824.5134-1-crosa@redhat.com>
> 
> Philippe Mathieu-Daudé (2):
>   BootLinuxConsoleTest: Let extract_from_deb handle various compressions
>   Boot Linux Console Test: add a test for the Raspberry Pi 2
> 
>  tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++--
>  1 file changed, 32 insertions(+), 2 deletions(-)

I was hoping this series would be included in "Acceptance Tests: target
architecture support" by Eduardo (removing the 'print' debug line) but I
just realized I forgot to Cc him :S

Re: [Qemu-devel] [PATCH v2 0/2] Acceptance Tests: Test the Raspberry Pi 2 board

[Philippe Mathieu-Daudé]

On 3/13/19 12:45 AM, Philippe Mathieu-Daudé wrote:
> Add the raspi2 to the list of boards tested by the
> boot_linux_console Avocado test.
> 
> Based on "Acceptance Tests: target architecture support" v5:
> https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04111.html
> Based-on: <20190312171824.5134-1-crosa@redhat.com>
> 
> Philippe Mathieu-Daudé (2):
>   BootLinuxConsoleTest: Let extract_from_deb handle various compressions
>   Boot Linux Console Test: add a test for the Raspberry Pi 2
> 
>  tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++--
>  1 file changed, 32 insertions(+), 2 deletions(-)

I was hoping this series would be included in "Acceptance Tests: target
architecture support" by Eduardo (removing the 'print' debug line) but I
just realized I forgot to Cc him :S

Re: [Qemu-devel] [PATCH v2 0/2] Acceptance Tests: Test the Raspberry Pi 2 board

[Eduardo Habkost]

On Sat, May 04, 2019 at 01:52:51PM +0200, Philippe Mathieu-Daudé wrote:
> On 3/13/19 12:45 AM, Philippe Mathieu-Daudé wrote:
> > Add the raspi2 to the list of boards tested by the
> > boot_linux_console Avocado test.
> > 
> > Based on "Acceptance Tests: target architecture support" v5:
> > https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04111.html
> > Based-on: <20190312171824.5134-1-crosa@redhat.com>
> > 
> > Philippe Mathieu-Daudé (2):
> >   BootLinuxConsoleTest: Let extract_from_deb handle various compressions
> >   Boot Linux Console Test: add a test for the Raspberry Pi 2
> > 
> >  tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++--
> >  1 file changed, 32 insertions(+), 2 deletions(-)
> 
> I was hoping this series would be included in "Acceptance Tests: target
> architecture support" by Eduardo (removing the 'print' debug line) but I
> just realized I forgot to Cc him :S

Problem solved: queued on python-next (after removing the print
statement from patch 2/2).  :)

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 0/2] Acceptance Tests: Test the Raspberry Pi 2 board

[Philippe Mathieu-Daudé]

On 5/6/19 11:41 PM, Eduardo Habkost wrote:
> On Sat, May 04, 2019 at 01:52:51PM +0200, Philippe Mathieu-Daudé wrote:
>> On 3/13/19 12:45 AM, Philippe Mathieu-Daudé wrote:
>>> Add the raspi2 to the list of boards tested by the
>>> boot_linux_console Avocado test.
>>>
>>> Based on "Acceptance Tests: target architecture support" v5:
>>> https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04111.html
>>> Based-on: <20190312171824.5134-1-crosa@redhat.com>
>>>
>>> Philippe Mathieu-Daudé (2):
>>>   BootLinuxConsoleTest: Let extract_from_deb handle various compressions
>>>   Boot Linux Console Test: add a test for the Raspberry Pi 2
>>>
>>>  tests/acceptance/boot_linux_console.py | 34 ++++++++++++++++++++++++--
>>>  1 file changed, 32 insertions(+), 2 deletions(-)
>>
>> I was hoping this series would be included in "Acceptance Tests: target
>> architecture support" by Eduardo (removing the 'print' debug line) but I
>> just realized I forgot to Cc him :S
> 
> Problem solved: queued on python-next (after removing the print
> statement from patch 2/2).  :)

Thanks a lot!

Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

[Eduardo Habkost]

On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> board and verify the serial is working.
> 
> If ARM is a target being built, "make check-acceptance" will
> automatically include this test by the use of the "arch:arm" tags.
> 
> Alternatively, this test can be run using:
> 
>     $ avocado run -t arch:arm tests/acceptance
>     $ avocado run -t machine:raspi2 tests/acceptance
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

We're getting timeouts on travis-ci:
https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289

I don't know yet if the guest is hanging, or if we just need to
increase the timeout.  I could reproduce the timeout locally,
too.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

[Cleber Rosa]

----- Original Message -----
> From: "Eduardo Habkost" <ehabkost@redhat.com>
> To: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> Cc: qemu-devel@nongnu.org, "Cleber Rosa" <crosa@redhat.com>, "Peter Maydell" <peter.maydell@linaro.org>,
> qemu-arm@nongnu.org, "Philippe Mathieu-Daudé" <f4bug@amsat.org>, "Andrew Baumann" <Andrew.Baumann@microsoft.com>
> Sent: Wednesday, May 22, 2019 4:52:34 PM
> Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2
> 
> On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> > From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > 
> > Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> > board and verify the serial is working.
> > 
> > If ARM is a target being built, "make check-acceptance" will
> > automatically include this test by the use of the "arch:arm" tags.
> > 
> > Alternatively, this test can be run using:
> > 
> >     $ avocado run -t arch:arm tests/acceptance
> >     $ avocado run -t machine:raspi2 tests/acceptance
> > 
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> We're getting timeouts on travis-ci:
> https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289
> 
> I don't know yet if the guest is hanging, or if we just need to
> increase the timeout.  I could reproduce the timeout locally,
> too.
> 
> --
> Eduardo
> 

It may be related to:

 https://bugs.launchpad.net/qemu/+bug/1829779

And this is a proof that we urgently need to have a better
way of presenting/storing test artifacts.  The brief output
is nice when everything goes well, but makes the test results
close to useless once a failure happens.

Philippe showed us how GitLab allows CI jobs to preserve
artifacts, so maybe the solution is to move the loads there.

- Cleber.

Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

[Philippe Mathieu-Daudé]

On 5/22/19 11:05 PM, Cleber Rosa wrote:
> ----- Original Message -----
>> From: "Eduardo Habkost" <ehabkost@redhat.com>
>> On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
>>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>
>>> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
>>> board and verify the serial is working.
>>>
>>> If ARM is a target being built, "make check-acceptance" will
>>> automatically include this test by the use of the "arch:arm" tags.
>>>
>>> Alternatively, this test can be run using:
>>>
>>>     $ avocado run -t arch:arm tests/acceptance
>>>     $ avocado run -t machine:raspi2 tests/acceptance
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>
>> We're getting timeouts on travis-ci:
>> https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289
>>
>> I don't know yet if the guest is hanging, or if we just need to
>> increase the timeout.  I could reproduce the timeout locally,
>> too.

That's odd, I can't reproduce (this test is quicker than the following
test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).

My guess is network issues, since this test use a different mirror:
archive.raspberrypi.org

Gerd already raised this problem (timeout reached while fetching
artifacts) to Cleber.
Cleber, can we add test_setup() methods that use different timeouts?

Regards,

Phil.

> It may be related to:
> 
>  https://bugs.launchpad.net/qemu/+bug/1829779
> 
> And this is a proof that we urgently need to have a better
> way of presenting/storing test artifacts.  The brief output
> is nice when everything goes well, but makes the test results
> close to useless once a failure happens.
> 
> Philippe showed us how GitLab allows CI jobs to preserve
> artifacts, so maybe the solution is to move the loads there.
> 
> - Cleber.
> 

Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

[Cleber Rosa]

----- Original Message -----
> From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> To: "Cleber Rosa" <crosa@redhat.com>, "Eduardo Habkost" <ehabkost@redhat.com>
> Cc: qemu-devel@nongnu.org, "Peter Maydell" <peter.maydell@linaro.org>, qemu-arm@nongnu.org, "Philippe Mathieu-Daudé"
> <f4bug@amsat.org>, "Andrew Baumann" <Andrew.Baumann@microsoft.com>, "Gerd Hoffmann" <kraxel@redhat.com>
> Sent: Thursday, May 23, 2019 5:29:22 AM
> Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2
> 
> On 5/22/19 11:05 PM, Cleber Rosa wrote:
> > ----- Original Message -----
> >> From: "Eduardo Habkost" <ehabkost@redhat.com>
> >> On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> >>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>>
> >>> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> >>> board and verify the serial is working.
> >>>
> >>> If ARM is a target being built, "make check-acceptance" will
> >>> automatically include this test by the use of the "arch:arm" tags.
> >>>
> >>> Alternatively, this test can be run using:
> >>>
> >>>     $ avocado run -t arch:arm tests/acceptance
> >>>     $ avocado run -t machine:raspi2 tests/acceptance
> >>>
> >>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>
> >> We're getting timeouts on travis-ci:
> >> https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289
> >>
> >> I don't know yet if the guest is hanging, or if we just need to
> >> increase the timeout.  I could reproduce the timeout locally,
> >> too.
> 
> That's odd, I can't reproduce (this test is quicker than the following
> test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).
> 
> My guess is network issues, since this test use a different mirror:
> archive.raspberrypi.org
> 

It could be a network issue, it could be something else.  I think the very
first step, and I'd urge us to get that on master ASAP, is to show the
entire logs in CI, I mean:

---

diff --git a/.travis.yml b/.travis.yml
index 6fd89b3d91..fd8f6ca2d2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -225,7 +225,7 @@ matrix:
     # Acceptance (Functional) tests
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="make AVOCADO_SHOW=test check-acceptance"
       addons:
         apt:
           packages:

---

That way we can know for sure what's going on. 

> Gerd already raised this problem (timeout reached while fetching
> artifacts) to Cleber.
> Cleber, can we add test_setup() methods that use different timeouts?
> 

Not in a released Avocado version.  Interestingly enough, I wrote a PoC
that adds different timeouts to tearDown()[1], that can be used the same
way for setUp(), and it looks like Intel DAOS is already using those
patches[2].

So, I guess I can work on a non-PoC version for that.

- Cleber.

[1] - https://github.com/avocado-framework/avocado/pull/3076
[2] - https://github.com/daos-stack/daos/commit/084ec23461e7bd9b997d4b6f5e8095a4eaffc685

> Regards,
> 
> Phil.
> 
> > It may be related to:
> > 
> >  https://bugs.launchpad.net/qemu/+bug/1829779
> > 
> > And this is a proof that we urgently need to have a better
> > way of presenting/storing test artifacts.  The brief output
> > is nice when everything goes well, but makes the test results
> > close to useless once a failure happens.
> > 
> > Philippe showed us how GitLab allows CI jobs to preserve
> > artifacts, so maybe the solution is to move the loads there.
> > 
> > - Cleber.
> > 
> 

Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

[Eduardo Habkost]

On Thu, May 23, 2019 at 09:40:06AM -0400, Cleber Rosa wrote:
> ----- Original Message -----
> > From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> > To: "Cleber Rosa" <crosa@redhat.com>, "Eduardo Habkost" <ehabkost@redhat.com>
> > Cc: qemu-devel@nongnu.org, "Peter Maydell" <peter.maydell@linaro.org>, qemu-arm@nongnu.org, "Philippe Mathieu-Daudé"
> > <f4bug@amsat.org>, "Andrew Baumann" <Andrew.Baumann@microsoft.com>, "Gerd Hoffmann" <kraxel@redhat.com>
> > Sent: Thursday, May 23, 2019 5:29:22 AM
> > Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2
> > 
> > On 5/22/19 11:05 PM, Cleber Rosa wrote:
> > > ----- Original Message -----
> > >> From: "Eduardo Habkost" <ehabkost@redhat.com>
> > >> On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> > >>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > >>>
> > >>> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> > >>> board and verify the serial is working.
> > >>>
> > >>> If ARM is a target being built, "make check-acceptance" will
> > >>> automatically include this test by the use of the "arch:arm" tags.
> > >>>
> > >>> Alternatively, this test can be run using:
> > >>>
> > >>>     $ avocado run -t arch:arm tests/acceptance
> > >>>     $ avocado run -t machine:raspi2 tests/acceptance
> > >>>
> > >>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > >>
> > >> We're getting timeouts on travis-ci:
> > >> https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289
> > >>
> > >> I don't know yet if the guest is hanging, or if we just need to
> > >> increase the timeout.  I could reproduce the timeout locally,
> > >> too.
> > 
> > That's odd, I can't reproduce (this test is quicker than the following
> > test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).
> > 
> > My guess is network issues, since this test use a different mirror:
> > archive.raspberrypi.org
> > 
> 
> It could be a network issue, it could be something else.  I think the very
> first step, and I'd urge us to get that on master ASAP, is to show the
> entire logs in CI, I mean:
> 
> ---
> 
> diff --git a/.travis.yml b/.travis.yml
> index 6fd89b3d91..fd8f6ca2d2 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -225,7 +225,7 @@ matrix:
>      # Acceptance (Functional) tests
>      - env:
>          - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
> -        - TEST_CMD="make check-acceptance"
> +        - TEST_CMD="make AVOCADO_SHOW=test check-acceptance"
>        addons:
>          apt:
>            packages:

I have applied this change on python-next.  The failure can be
seen here:
https://travis-ci.org/ehabkost/qemu/jobs/541758418#L4033

While we don't fix this, I'm removing this test case from the
queue, so I can send a pull request.  Sorry for taking so long to
act on this.

-- 
Eduardo