[PATCH] test_sleep.py: make sleep time and margin configurable

[PATCH] test_sleep.py: make sleep time and margin configurable

[Heiko Schocher]

make the sleep time and the margin configurable.

Signed-off-by: Heiko Schocher <hs@denx.de>
---

travis build:
https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225

This patch is needed as I start test/py now within tbot [1]. On
some configurations U-Boot is compiled on a build machine for
example in munich, while the board under test is in my lab in
hungary.

So the 0.25 seconds default margin is often to low because
of latencies on the net.

See as an example configuration (within tbot):

https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29

[1] http://tbot.tools/modules/tc.html#u-boot-test-py

 test/py/tests/test_sleep.py | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py
index b69edf26ef..392af29db2 100644
--- a/test/py/tests/test_sleep.py
+++ b/test/py/tests/test_sleep.py
@@ -11,6 +11,12 @@ change test behavior.
 # Setup env__sleep_accurate to False if time is not accurate on your platform
 env__sleep_accurate = False
 
+# Setup env__sleep_time time in seconds board is set to sleep
+env__sleep_time = 3
+
+# Setup env__sleep_margin set a margin for any system overhead
+env__sleep_margin = 0.25
+
 """
 
 def test_sleep(u_boot_console):
@@ -23,13 +29,15 @@ def test_sleep(u_boot_console):
 
     if u_boot_console.config.buildconfig.get('config_cmd_misc', 'n') != 'y':
         pytest.skip('sleep command not supported')
+
     # 3s isn't too long, but is enough to cross a few second boundaries.
-    sleep_time = 3
+    sleep_time = u_boot_console.config.env.get('env__sleep_time', 3)
+    sleep_margin = u_boot_console.config.env.get('env__sleep_margin', 0.25)
     tstart = time.time()
     u_boot_console.run_command('sleep %d' % sleep_time)
     tend = time.time()
     elapsed = tend - tstart
     assert elapsed >= (sleep_time - 0.01)
     if not u_boot_console.config.gdbserver:
-        # 0.25s margin is hopefully enough to account for any system overhead.
-        assert elapsed < (sleep_time + 0.25)
+        # margin is hopefully enough to account for any system overhead.
+        assert elapsed < (sleep_time + sleep_margin)
-- 
2.24.1

[PATCH] test_sleep.py: make sleep time and margin configurable

[Simon Glass]

On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs@denx.de> wrote:
>
> make the sleep time and the margin configurable.
>
> Signed-off-by: Heiko Schocher <hs@denx.de>
> ---
>
> travis build:
> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
>
> This patch is needed as I start test/py now within tbot [1]. On
> some configurations U-Boot is compiled on a build machine for
> example in munich, while the board under test is in my lab in
> hungary.
>
> So the 0.25 seconds default margin is often to low because
> of latencies on the net.
>
> See as an example configuration (within tbot):
>
> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
>
> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
>
>  test/py/tests/test_sleep.py | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>

Related, at some point we should change sandbox to fake the time
movement since this test currently waits for three seconds even on
sandbox.

[PATCH] test_sleep.py: make sleep time and margin configurable

[Stephen Warren]

On 6/4/20 9:24 AM, Heiko Schocher wrote:
> make the sleep time and the margin configurable.

Reviewed-by: Stephen Warren <swarren@nvidia.com>

[PATCH] test_sleep.py: make sleep time and margin configurable

[Stephen Warren]

On 6/7/20 7:45 AM, Simon Glass wrote:
> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs@denx.de> wrote:
>>
>> make the sleep time and the margin configurable.
>>
>> Signed-off-by: Heiko Schocher <hs@denx.de>
>> ---
>>
>> travis build:
>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
>>
>> This patch is needed as I start test/py now within tbot [1]. On
>> some configurations U-Boot is compiled on a build machine for
>> example in munich, while the board under test is in my lab in
>> hungary.
>>
>> So the 0.25 seconds default margin is often to low because
>> of latencies on the net.
>>
>> See as an example configuration (within tbot):
>>
>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
>>
>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
>>
>>  test/py/tests/test_sleep.py | 14 +++++++++++---
>>  1 file changed, 11 insertions(+), 3 deletions(-)
> 
> Reviewed-by: Simon Glass <sjg@chromium.org>
> 
> Related, at some point we should change sandbox to fake the time
> movement since this test currently waits for three seconds even on
> sandbox.

We definitely shouldn't do that; that's the exact kind of failure this
test is intended to detect.

[PATCH] test_sleep.py: make sleep time and margin configurable

[Simon Glass]

Hi Stephen,

On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren@wwwdotorg.org> wrote:
>
> On 6/7/20 7:45 AM, Simon Glass wrote:
> > On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs@denx.de> wrote:
> >>
> >> make the sleep time and the margin configurable.
> >>
> >> Signed-off-by: Heiko Schocher <hs@denx.de>
> >> ---
> >>
> >> travis build:
> >> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
> >>
> >> This patch is needed as I start test/py now within tbot [1]. On
> >> some configurations U-Boot is compiled on a build machine for
> >> example in munich, while the board under test is in my lab in
> >> hungary.
> >>
> >> So the 0.25 seconds default margin is often to low because
> >> of latencies on the net.
> >>
> >> See as an example configuration (within tbot):
> >>
> >> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
> >>
> >> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
> >>
> >>  test/py/tests/test_sleep.py | 14 +++++++++++---
> >>  1 file changed, 11 insertions(+), 3 deletions(-)
> >
> > Reviewed-by: Simon Glass <sjg@chromium.org>
> >
> > Related, at some point we should change sandbox to fake the time
> > movement since this test currently waits for three seconds even on
> > sandbox.
>
> We definitely shouldn't do that; that's the exact kind of failure this
> test is intended to detect.

No, we're not looking for bugs in sandbox's time handling. We are just
testing the plumbing associated with delaying (timer driver, etc.).

Regards,
Simon

[PATCH] test_sleep.py: make sleep time and margin configurable

[Stephen Warren]

On 6/8/20 11:12 AM, Simon Glass wrote:
> Hi Stephen,
> 
> On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>
>> On 6/7/20 7:45 AM, Simon Glass wrote:
>>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs@denx.de> wrote:
>>>>
>>>> make the sleep time and the margin configurable.
>>>>
>>>> Signed-off-by: Heiko Schocher <hs@denx.de>
>>>> ---
>>>>
>>>> travis build:
>>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
>>>>
>>>> This patch is needed as I start test/py now within tbot [1]. On
>>>> some configurations U-Boot is compiled on a build machine for
>>>> example in munich, while the board under test is in my lab in
>>>> hungary.
>>>>
>>>> So the 0.25 seconds default margin is often to low because
>>>> of latencies on the net.
>>>>
>>>> See as an example configuration (within tbot):
>>>>
>>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
>>>>
>>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
>>>>
>>>>  test/py/tests/test_sleep.py | 14 +++++++++++---
>>>>  1 file changed, 11 insertions(+), 3 deletions(-)
>>>
>>> Reviewed-by: Simon Glass <sjg@chromium.org>
>>>
>>> Related, at some point we should change sandbox to fake the time
>>> movement since this test currently waits for three seconds even on
>>> sandbox.
>>
>> We definitely shouldn't do that; that's the exact kind of failure this
>> test is intended to detect.
> 
> No, we're not looking for bugs in sandbox's time handling. We are just
> testing the plumbing associated with delaying (timer driver, etc.).

The entire purpose of the test is to look for bugs in the backend
implementation of the time handling. I should know; I wrote the test!

[PATCH] test_sleep.py: make sleep time and margin configurable

[Simon Glass]

Hi Stephen,

On Mon, 8 Jun 2020 at 11:25, Stephen Warren <swarren@wwwdotorg.org> wrote:
>
> On 6/8/20 11:12 AM, Simon Glass wrote:
> > Hi Stephen,
> >
> > On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren@wwwdotorg.org> wrote:
> >>
> >> On 6/7/20 7:45 AM, Simon Glass wrote:
> >>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs@denx.de> wrote:
> >>>>
> >>>> make the sleep time and the margin configurable.
> >>>>
> >>>> Signed-off-by: Heiko Schocher <hs@denx.de>
> >>>> ---
> >>>>
> >>>> travis build:
> >>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
> >>>>
> >>>> This patch is needed as I start test/py now within tbot [1]. On
> >>>> some configurations U-Boot is compiled on a build machine for
> >>>> example in munich, while the board under test is in my lab in
> >>>> hungary.
> >>>>
> >>>> So the 0.25 seconds default margin is often to low because
> >>>> of latencies on the net.
> >>>>
> >>>> See as an example configuration (within tbot):
> >>>>
> >>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
> >>>>
> >>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
> >>>>
> >>>>  test/py/tests/test_sleep.py | 14 +++++++++++---
> >>>>  1 file changed, 11 insertions(+), 3 deletions(-)
> >>>
> >>> Reviewed-by: Simon Glass <sjg@chromium.org>
> >>>
> >>> Related, at some point we should change sandbox to fake the time
> >>> movement since this test currently waits for three seconds even on
> >>> sandbox.
> >>
> >> We definitely shouldn't do that; that's the exact kind of failure this
> >> test is intended to detect.
> >
> > No, we're not looking for bugs in sandbox's time handling. We are just
> > testing the plumbing associated with delaying (timer driver, etc.).
>
> The entire purpose of the test is to look for bugs in the backend
> implementation of the time handling. I should know; I wrote the test!

OK and we have time_ut.c as well.

Perhaps we should disable this test on sandbox then? It really doesn't
make sense to be testing timeouts on sandbox IMO and it costs us 3
seconds on each test run.

Regards,
Simon

[PATCH] test_sleep.py: make sleep time and margin configurable

[Stephen Warren]

On 6/16/20 9:11 PM, Simon Glass wrote:
> Hi Stephen,
> 
> On Mon, 8 Jun 2020 at 11:25, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>
>> On 6/8/20 11:12 AM, Simon Glass wrote:
>>> Hi Stephen,
>>>
>>> On Mon, 8 Jun 2020 at 10:43, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>>
>>>> On 6/7/20 7:45 AM, Simon Glass wrote:
>>>>> On Thu, 4 Jun 2020 at 09:24, Heiko Schocher <hs@denx.de> wrote:
>>>>>>
>>>>>> make the sleep time and the margin configurable.
>>>>>>
>>>>>> Signed-off-by: Heiko Schocher <hs@denx.de>
>>>>>> ---
>>>>>>
>>>>>> travis build:
>>>>>> https://travis-ci.org/github/hsdenx/u-boot-test/builds/694545225
>>>>>>
>>>>>> This patch is needed as I start test/py now within tbot [1]. On
>>>>>> some configurations U-Boot is compiled on a build machine for
>>>>>> example in munich, while the board under test is in my lab in
>>>>>> hungary.
>>>>>>
>>>>>> So the 0.25 seconds default margin is often to low because
>>>>>> of latencies on the net.
>>>>>>
>>>>>> See as an example configuration (within tbot):
>>>>>>
>>>>>> https://github.com/EmbLux-Kft/tbot-tbot2go/blob/devel/boards/aristainetos.py#L29
>>>>>>
>>>>>> [1] http://tbot.tools/modules/tc.html#u-boot-test-py
>>>>>>
>>>>>>  test/py/tests/test_sleep.py | 14 +++++++++++---
>>>>>>  1 file changed, 11 insertions(+), 3 deletions(-)
>>>>>
>>>>> Reviewed-by: Simon Glass <sjg@chromium.org>
>>>>>
>>>>> Related, at some point we should change sandbox to fake the time
>>>>> movement since this test currently waits for three seconds even on
>>>>> sandbox.
>>>>
>>>> We definitely shouldn't do that; that's the exact kind of failure this
>>>> test is intended to detect.
>>>
>>> No, we're not looking for bugs in sandbox's time handling. We are just
>>> testing the plumbing associated with delaying (timer driver, etc.).
>>
>> The entire purpose of the test is to look for bugs in the backend
>> implementation of the time handling. I should know; I wrote the test!
> 
> OK and we have time_ut.c as well.
> 
> Perhaps we should disable this test on sandbox then? It really doesn't
> make sense to be testing timeouts on sandbox IMO and it costs us 3
> seconds on each test run.

I'd rather keep such policy out of the test framework. If you personally
don't want to run the test, you can always specify '-k not sleep' (if I
got the syntax right...) when running test.py. I think e.g. the Travis
test configuration might already do this, as an example.

[PATCH] test_sleep.py: make sleep time and margin configurable

[Tom Rini]

On Thu, Jun 04, 2020 at 05:24:00PM +0200, Heiko Schocher wrote:

> make the sleep time and the margin configurable.
> 
> Signed-off-by: Heiko Schocher <hs@denx.de>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Stephen Warren <swarren@nvidia.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200707/689dfb99/attachment.sig>