[LTP] [PATCH] open_posix/timer_getoverrun/2-3: Fix test for systems with low timer precision

[LTP] [PATCH] open_posix/timer_getoverrun/2-3: Fix test for systems with low timer precision

[Joerg Vehlow]

From: Joerg Vehlow <joerg.vehlow@aox-tech.de>

On systems with low timer precision the test always fails, because the allowed
maximum number of overruns is calculated from the expected overruns + 10%.
If the expected overruns is less than 200, there is no tollerance.
This happens, if the precision of the timer is less than or equal to 4ms.
E.g. qemu-arm64 without high resolution timer the accuracy is only 4ms.

Signed-off-by: Joerg Vehlow <joerg.vehlow@aox-tech.de>
---
 .../conformance/interfaces/timer_getoverrun/2-3.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
index 96b7d01e6..3df3a9f01 100644
--- a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
+++ b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
@@ -55,6 +55,7 @@ int main(void)
 	int overruns;
 	int valuensec, intervalnsec, expectedoverruns;
 	int fudge;
+	int duration;
 
 	if (sigemptyset(&set) != 0) {
 		perror("sigemptyset() did not return success\n");
@@ -94,11 +95,15 @@ int main(void)
 
 	valuensec = tsres.tv_nsec;
 	intervalnsec = 2 * valuensec;
-	//expectedoverruns = (1000000000 - valuensec) / intervalnsec;
-	expectedoverruns = 1000000000 / intervalnsec - 1;
+	expectedoverruns = 0;
+	duration = 0;
+	while (expectedoverruns < 1000) {
+		duration++;
+		expectedoverruns = duration * (1000000000 / intervalnsec - 1);
+	}
 
-	printf("value = %d sec, interval = %d nsec, "
-	       "expected overruns = %d\n", 1, intervalnsec, expectedoverruns);
+	printf("duration = %d sec, interval = %d nsec, "
+	       "expected overruns = %d\n", duration, intervalnsec, expectedoverruns);
 
 	its.it_interval.tv_sec = 0;
 	its.it_interval.tv_nsec = intervalnsec;
@@ -113,7 +118,7 @@ int main(void)
 	}
 	//tssleep.tv_nsec = valuensec + (expectedoverruns*intervalnsec);
 	tssleep.tv_nsec = 0;
-	tssleep.tv_sec = 2;
+	tssleep.tv_sec = duration + 1;
 	if (nanosleep(&tssleep, NULL) != 0) {
 		perror("nanosleep() did not return success\n");
 		return PTS_UNRESOLVED;
-- 
2.20.1

[LTP] [PATCH] open_posix/timer_getoverrun/2-3: Fix test for systems with low timer precision

[Jan Stancek]

----- Original Message -----
> From: Joerg Vehlow <joerg.vehlow@aox-tech.de>
> 

Hi,

> On systems with low timer precision the test always fails, because the
> allowed
> maximum number of overruns is calculated from the expected overruns + 10%.

Did you mean 1% here?
  fudge = expectedoverruns / 100;

> If the expected overruns is less than 200, there is no tollerance.
> This happens, if the precision of the timer is less than or equal to 4ms.
> E.g. qemu-arm64 without high resolution timer the accuracy is only 4ms.

Would tweaking tolerance work too? E.g. use float, round up.

> 
> Signed-off-by: Joerg Vehlow <joerg.vehlow@aox-tech.de>
> ---
>  .../conformance/interfaces/timer_getoverrun/2-3.c | 15 ++++++++++-----
>  1 file changed, 10 insertions(+), 5 deletions(-)
> 
> diff --git
> a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> index 96b7d01e6..3df3a9f01 100644
> ---
> a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> +++
> b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> @@ -55,6 +55,7 @@ int main(void)
>  	int overruns;
>  	int valuensec, intervalnsec, expectedoverruns;
>  	int fudge;
> +	int duration;
>  
>  	if (sigemptyset(&set) != 0) {
>  		perror("sigemptyset() did not return success\n");
> @@ -94,11 +95,15 @@ int main(void)
>  
>  	valuensec = tsres.tv_nsec;
>  	intervalnsec = 2 * valuensec;
> -	//expectedoverruns = (1000000000 - valuensec) / intervalnsec;
> -	expectedoverruns = 1000000000 / intervalnsec - 1;
> +	expectedoverruns = 0;
> +	duration = 0;
> +	while (expectedoverruns < 1000) {
> +		duration++;
> +		expectedoverruns = duration * (1000000000 / intervalnsec - 1);

I was assuming -1 in original code is to cope with final timer expiration of tssleep.tv_sec,
which might not be counted as "overrun". What does the -1 do in your formula?
Why is it inside brackets?

When I try to force different interval values, it fails for me (on x86):
3ms
# ./timer_getoverrun_2-3.run-test 
duration = 7 sec, interval = 6000000 nsec, expected overruns = 1155
1166 overruns occurred
FAIL:  1166 overruns sent; expected 1155

5ms
# ./timer_getoverrun_2-3.run-test 
duration = 11 sec, interval = 10000000 nsec, expected overruns = 1089
1100 overruns occurred
FAIL:  1100 overruns sent; expected 1089

Regards,
Jan

[LTP] [PATCH] open_posix/timer_getoverrun/2-3: Fix test for systems with low timer precision

[Joerg Vehlow]

Hi,
>> On systems with low timer precision the test always fails, because the
>> allowed
>> maximum number of overruns is calculated from the expected overruns + 10%.
> Did you mean 1% here?
>    fudge = expectedoverruns / 100;
Of course, you are right
>
>> If the expected overruns is less than 200, there is no tollerance.
>> This happens, if the precision of the timer is less than or equal to 4ms.
>> E.g. qemu-arm64 without high resolution timer the accuracy is only 4ms.
> Would tweaking tolerance work too? E.g. use float, round up.
Actually rounding up would not work. I tried it with a timer accuracy of 
4ms and I needed
at least a fudge of two. Rounding up would only result in one. That it 
why I decided to
go for extending the duration
>
> I was assuming -1 in original code is to cope with final timer expiration of tssleep.tv_sec,
> which might not be counted as "overrun". What does the -1 do in your formula?
> Why is it inside brackets?
You may be right. I am not completely sure what it is used for, but I 
guess you are right.
>
> When I try to force different interval values, it fails for me (on x86):
> 3ms
> # ./timer_getoverrun_2-3.run-test
> duration = 7 sec, interval = 6000000 nsec, expected overruns = 1155
> 1166 overruns occurred
> FAIL:  1166 overruns sent; expected 1155
>
> 5ms
> # ./timer_getoverrun_2-3.run-test
> duration = 11 sec, interval = 10000000 nsec, expected overruns = 1089
> 1100 overruns occurred
> FAIL:  1100 overruns sent; expected 1089
How is it possible to force the timer resolution? I tested on qemu
with aarch64 and x86_64 with and without highres=off and ir worked.

If the -1 is outside the brackets, it would fix both of your test cases.
(1155 + 7 - 1)? = 1161; +1% = 1172 -> expected overruns [1161, 1172)
(1089 + 11 - 1) = 1099; +1% = 1109 ->expected overruns [1099, 1109)

J?rg

[LTP] [PATCH] open_posix/timer_getoverrun/2-3: Fix test for systems with low timer precision

[Jan Stancek]

On Wed, Oct 16, 2019 at 07:39:52AM +0200, Joerg Vehlow wrote:
>Actually rounding up would not work. I tried it with a timer accuracy 
>of 4ms and I needed
>at least a fudge of two. Rounding up would only result in one. That 
>it why I decided to
>go for extending the duration
>>
>>I was assuming -1 in original code is to cope with final timer expiration of tssleep.tv_sec,
>>which might not be counted as "overrun". What does the -1 do in your formula?
>>Why is it inside brackets?
>You may be right. I am not completely sure what it is used for, but I 
>guess you are right.
>>
>>When I try to force different interval values, it fails for me (on x86):
>>3ms
>># ./timer_getoverrun_2-3.run-test
>>duration = 7 sec, interval = 6000000 nsec, expected overruns = 1155
>>1166 overruns occurred
>>FAIL:  1166 overruns sent; expected 1155
>>
>>5ms
>># ./timer_getoverrun_2-3.run-test
>>duration = 11 sec, interval = 10000000 nsec, expected overruns = 1089
>>1100 overruns occurred
>>FAIL:  1100 overruns sent; expected 1089
>How is it possible to force the timer resolution? I tested on qemu
>with aarch64 and x86_64 with and without highres=off and ir worked.

I didn't change timer resolution, I only used larger interval values
as multiple of timer resolution:
   intervalnsec = largeX * timer_resolution

I'd prefer we tweak the tolerance rather than make test run longer.
I'm thinking just allow ~50ms of extra overruns, and don't be so
strict about absolute number of overruns. (KVM guests and s390 lpars
tend to suffer from higher steal time).

diff --git a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
index 96b7d01e6ffe..66f8b583a5a6 100644
--- a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
+++ b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
@@ -94,11 +94,17 @@ int main(void)
  
         valuensec = tsres.tv_nsec;
         intervalnsec = 2 * valuensec;
-       //expectedoverruns = (1000000000 - valuensec) / intervalnsec;
         expectedoverruns = 1000000000 / intervalnsec - 1;
  
+       /*
+        * waking up from sleep isn't instant, we can overshoot.
+        * Allow up to ~50ms worth of extra overruns.
+        */
+       fudge = 50000000 / intervalnsec + 1;
+
         printf("value = %d sec, interval = %d nsec, "
-              "expected overruns = %d\n", 1, intervalnsec, expectedoverruns);
+              "expected overruns = %d, fudge = %d\n", 1,
+              intervalnsec, expectedoverruns, fudge);
  
         its.it_interval.tv_sec = 0;
         its.it_interval.tv_nsec = intervalnsec;
@@ -146,7 +152,6 @@ int main(void)
          * extra expiries after the nanosleep completes so do
          * a range check.
          */
-       fudge = expectedoverruns / 100;
         if (overruns >= expectedoverruns && overruns < expectedoverruns + fudge) {
                 printf("Test PASSED\n");
                 return PTS_PASS;

[LTP] [PATCH] open_posix/timer_getoverrun/2-3: Fix test for systems with low timer precision

[Joerg Vehlow]

> I didn't change timer resolution, I only used larger interval values
> as multiple of timer resolution:
> ? intervalnsec = largeX * timer_resolution
>
ah ok
> I'd prefer we tweak the tolerance rather than make test run longer.
> I'm thinking just allow ~50ms of extra overruns, and don't be so
> strict about absolute number of overruns. (KVM guests and s390 lpars
> tend to suffer from higher steal time).
>
> diff --git 
> a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c 
> b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c 
>
> index 96b7d01e6ffe..66f8b583a5a6 100644
> --- 
> a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> +++ 
> b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> @@ -94,11 +94,17 @@ int main(void)
>
> ??????? valuensec = tsres.tv_nsec;
> ??????? intervalnsec = 2 * valuensec;
> -?????? //expectedoverruns = (1000000000 - valuensec) / intervalnsec;
> ??????? expectedoverruns = 1000000000 / intervalnsec - 1;
>
> +?????? /*
> +??????? * waking up from sleep isn't instant, we can overshoot.
> +??????? * Allow up to ~50ms worth of extra overruns.
> +??????? */
> +?????? fudge = 50000000 / intervalnsec + 1;
> +
> ??????? printf("value = %d sec, interval = %d nsec, "
> -????????????? "expected overruns = %d\n", 1, intervalnsec, 
> expectedoverruns);
> +????????????? "expected overruns = %d, fudge = %d\n", 1,
> +????????????? intervalnsec, expectedoverruns, fudge);
>
> ??????? its.it_interval.tv_sec = 0;
> ??????? its.it_interval.tv_nsec = intervalnsec;
> @@ -146,7 +152,6 @@ int main(void)
> ???????? * extra expiries after the nanosleep completes so do
> ???????? * a range check.
> ???????? */
> -?????? fudge = expectedoverruns / 100;
> ??????? if (overruns >= expectedoverruns && overruns < 
> expectedoverruns + fudge) {
> ??????????????? printf("Test PASSED\n");
> ??????????????? return PTS_PASS;
>
I wonder if there was a way to do this more accurate, like busy waiting 
while signals
are blocked. But I would also be fine with your solution.

[LTP] [PATCH] timer_getoverrun/2-3: increase tolerance for overshoot

[Jan Stancek]

Joerg reports that test fails on systems with low timer precision (qemu-arm64,
4ms timer precision) due to way we calculate maximum number of overruns.
If the expected overruns is less than 200, there is no tollerance.

Allow up to ~50ms of extra overruns.

Reported-by: Joerg Vehlow <joerg.vehlow@aox-tech.de>
Acked-by: Joerg Vehlow <joerg.vehlow@aox-tech.de>
Signed-off-by: Jan Stancek <jstancek@redhat.com>
---
 .../conformance/interfaces/timer_getoverrun/2-3.c             | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
index 96b7d01e6ffe..66f8b583a5a6 100644
--- a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
+++ b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
@@ -94,11 +94,17 @@ int main(void)
 
 	valuensec = tsres.tv_nsec;
 	intervalnsec = 2 * valuensec;
-	//expectedoverruns = (1000000000 - valuensec) / intervalnsec;
 	expectedoverruns = 1000000000 / intervalnsec - 1;
 
+	/*
+	 * waking up from sleep isn't instant, we can overshoot.
+	 * Allow up to ~50ms worth of extra overruns.
+	 */
+	fudge = 50000000 / intervalnsec + 1;
+
 	printf("value = %d sec, interval = %d nsec, "
-	       "expected overruns = %d\n", 1, intervalnsec, expectedoverruns);
+	       "expected overruns = %d, fudge = %d\n", 1,
+	       intervalnsec, expectedoverruns, fudge);
 
 	its.it_interval.tv_sec = 0;
 	its.it_interval.tv_nsec = intervalnsec;
@@ -146,7 +152,6 @@ int main(void)
 	 * extra expiries after the nanosleep completes so do
 	 * a range check.
 	 */
-	fudge = expectedoverruns / 100;
 	if (overruns >= expectedoverruns && overruns < expectedoverruns + fudge) {
 		printf("Test PASSED\n");
 		return PTS_PASS;
-- 
1.8.3.1

[LTP] [PATCH] timer_getoverrun/2-3: increase tolerance for overshoot

[Cyril Hrubis]

Hi!
> Joerg reports that test fails on systems with low timer precision (qemu-arm64,
> 4ms timer precision) due to way we calculate maximum number of overruns.
> If the expected overruns is less than 200, there is no tollerance.
> 
> Allow up to ~50ms of extra overruns.
> 
> Reported-by: Joerg Vehlow <joerg.vehlow@aox-tech.de>
> Acked-by: Joerg Vehlow <joerg.vehlow@aox-tech.de>
> Signed-off-by: Jan Stancek <jstancek@redhat.com>
> ---
>  .../conformance/interfaces/timer_getoverrun/2-3.c             | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> index 96b7d01e6ffe..66f8b583a5a6 100644
> --- a/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> +++ b/testcases/open_posix_testsuite/conformance/interfaces/timer_getoverrun/2-3.c
> @@ -94,11 +94,17 @@ int main(void)
>  
>  	valuensec = tsres.tv_nsec;
>  	intervalnsec = 2 * valuensec;
> -	//expectedoverruns = (1000000000 - valuensec) / intervalnsec;
>  	expectedoverruns = 1000000000 / intervalnsec - 1;
>  
> +	/*
> +	 * waking up from sleep isn't instant, we can overshoot.
> +	 * Allow up to ~50ms worth of extra overruns.
> +	 */
> +	fudge = 50000000 / intervalnsec + 1;

I wonder if we can measure the sleep with monotonic clock, then we can
be more precise, since we would know how much we overshoot. But it's
probably not worth of complicating the code.

Either way I'm fine with this version as well, acked.

>  	printf("value = %d sec, interval = %d nsec, "
> -	       "expected overruns = %d\n", 1, intervalnsec, expectedoverruns);
> +	       "expected overruns = %d, fudge = %d\n", 1,
> +	       intervalnsec, expectedoverruns, fudge);
>  
>  	its.it_interval.tv_sec = 0;
>  	its.it_interval.tv_nsec = intervalnsec;
> @@ -146,7 +152,6 @@ int main(void)
>  	 * extra expiries after the nanosleep completes so do
>  	 * a range check.
>  	 */
> -	fudge = expectedoverruns / 100;
>  	if (overruns >= expectedoverruns && overruns < expectedoverruns + fudge) {
>  		printf("Test PASSED\n");
>  		return PTS_PASS;
> -- 
> 1.8.3.1
> 
> 
> -- 
> Mailing list info: https://lists.linux.it/listinfo/ltp

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH] timer_getoverrun/2-3: increase tolerance for overshoot

[Jan Stancek]

----- Original Message -----
> Hi!
> I wonder if we can measure the sleep with monotonic clock, then we can
> be more precise, since we would know how much we overshoot. But it's
> probably not worth of complicating the code.
> 
> Either way I'm fine with this version as well, acked.

I pushed current patch. If we find it's still causing issues
I'll look into the idea with monotonic clock.

Thanks,
Jan

[LTP] [PATCH] timer_getoverrun/2-3: increase tolerance for overshoot

[Joerg Vehlow]

Hi,

Am 18.10.2019 um 09:58 schrieb Jan Stancek:
> ----- Original Message -----
>> Hi!
>> I wonder if we can measure the sleep with monotonic clock, then we can
>> be more precise, since we would know how much we overshoot. But it's
>> probably not worth of complicating the code.
>>
>> Either way I'm fine with this version as well, acked.
> I pushed current patch. If we find it's still causing issues
> I'll look into the idea with monotonic clock.
>
Thanks for fixing this. I'd like to mention, that CLOCK_MONOTONIC is an
optionalfeature in posix. I do not know if there are systems missing it,
that might run these test.

J?rg