* [PATCH 0/2] selftests: bump timeout for firmware and kmod @ 2023-02-06 23:43 Luis Chamberlain 2023-02-06 23:43 ` [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 Luis Chamberlain 2023-02-06 23:43 ` [PATCH 2/2] selftests/firmware: increase timeout from 165 to 230 Luis Chamberlain 0 siblings, 2 replies; 9+ messages in thread From: Luis Chamberlain @ 2023-02-06 23:43 UTC (permalink / raw) To: shuah, linux-kselftest Cc: gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel, Luis Chamberlain Shuah, I'd like this to go through your tree as this is timeout related. In order to help me help developers run tests against the components I maintain much easily I have enabled selftests support on kdevops [0]. kdevops deals with abstractsions like letting you pick virtualization or cloud solutions to run the tests using kconfig, installs all dependencies for you, and with just a few make target commands can get you the latest linux-next tested against selftests. If other find this useful and would like support for their selftests on kdevops feel free to send patches. Eventually the idea is to be able to run as many selftests in parallel using different guests for each main selftest to speed up tests. Prior to this I used to run tests manually, now the selftests helpers are used (./tools/testing/selftests/run_kselftest.sh -s) and with this the default selftest timeout is hit. This just increases that for the few selftests I help maintain where obviously its not enough anymore. Note: on the firmware side I am spotting an OOM triggered by running tests in a loop, so far I hit in the android configuration but its not clear if the issue is just for that setup. [0] https://github.com/linux-kdevops/kdevops Luis Chamberlain (2): selftests/kmod: increase the kmod timeout from 45 to 165 selftests/firmware: increase timeout from 165 to 230 tools/testing/selftests/firmware/settings | 8 +++++++- tools/testing/selftests/kmod/settings | 4 ++++ 2 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kmod/settings -- 2.39.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 2023-02-06 23:43 [PATCH 0/2] selftests: bump timeout for firmware and kmod Luis Chamberlain @ 2023-02-06 23:43 ` Luis Chamberlain 2023-02-27 22:32 ` Shuah Khan 2023-02-06 23:43 ` [PATCH 2/2] selftests/firmware: increase timeout from 165 to 230 Luis Chamberlain 1 sibling, 1 reply; 9+ messages in thread From: Luis Chamberlain @ 2023-02-06 23:43 UTC (permalink / raw) To: shuah, linux-kselftest Cc: gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel, Luis Chamberlain The default sefltests timeout is 45 seconds. If you run the kmod selftests on your own with say: ./tools/testings/selftests/kmod.sh Then the default timeout won't be in effect. I've never ran kmod selftests using the generic make wrapper (./tools/testing/selftests/run_kselftest.sh -s) util now that I have support for it on kdevops [0]. And with that the test is limitted to the default timeout which we quickly run into. Bump this up to what I see is required on 8GiB / 8 vcpu libvirt q35 guest as can be easily created now with kdevops. To run selftests with kdevops: make menuconfig # enable dedicated selftests and kmod test make make bringup make linux make selftests-kmod This ends up taking about 280 seconds now, give or take add 50 seconds more more and we end up with 350. Document the rationale. [0] https://github.com/linux-kdevops/kdevops Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> --- tools/testing/selftests/kmod/settings | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 tools/testing/selftests/kmod/settings diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings new file mode 100644 index 000000000000..6fca0f1a4594 --- /dev/null +++ b/tools/testing/selftests/kmod/settings @@ -0,0 +1,4 @@ +# measured from a manual run: +# time ./tools/testing/selftests/kmod/kmod.sh +# Then add ~50 seconds more gracetime. +timeout=350 -- 2.39.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 2023-02-06 23:43 ` [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 Luis Chamberlain @ 2023-02-27 22:32 ` Shuah Khan 2023-02-27 22:42 ` Luis Chamberlain 0 siblings, 1 reply; 9+ messages in thread From: Shuah Khan @ 2023-02-27 22:32 UTC (permalink / raw) To: Luis Chamberlain, shuah, linux-kselftest Cc: gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel, Shuah Khan On 2/6/23 16:43, Luis Chamberlain wrote: > The default sefltests timeout is 45 seconds. If you run the kmod > selftests on your own with say: > > ./tools/testings/selftests/kmod.sh > > Then the default timeout won't be in effect. > > I've never ran kmod selftests using the generic make wrapper > (./tools/testing/selftests/run_kselftest.sh -s) util now > that I have support for it on kdevops [0]. And with that the > test is limitted to the default timeout which we quickly run > into. Bump this up to what I see is required on 8GiB / 8 vcpu > libvirt q35 guest as can be easily created now with kdevops. > > To run selftests with kdevops: > > make menuconfig # enable dedicated selftests and kmod test > make > make bringup > make linux > make selftests-kmod > > This ends up taking about 280 seconds now, give or take add > 50 seconds more more and we end up with 350. Document the > rationale. > > [0] https://github.com/linux-kdevops/kdevops > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> > --- > tools/testing/selftests/kmod/settings | 4 ++++ > 1 file changed, 4 insertions(+) > create mode 100644 tools/testing/selftests/kmod/settings > > diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings > new file mode 100644 > index 000000000000..6fca0f1a4594 > --- /dev/null > +++ b/tools/testing/selftests/kmod/settings > @@ -0,0 +1,4 @@ > +# measured from a manual run: > +# time ./tools/testing/selftests/kmod/kmod.sh > +# Then add ~50 seconds more gracetime. > +timeout=350 Adding timeouts like this for individual tests increases the overall kselftest run-time. I am not in favor of adding timeouts. We have to find a better way to do this. thanks, -- Shuah ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 2023-02-27 22:32 ` Shuah Khan @ 2023-02-27 22:42 ` Luis Chamberlain 2023-03-03 20:35 ` Shuah Khan 0 siblings, 1 reply; 9+ messages in thread From: Luis Chamberlain @ 2023-02-27 22:42 UTC (permalink / raw) To: Shuah Khan Cc: shuah, linux-kselftest, gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel On Mon, Feb 27, 2023 at 03:32:50PM -0700, Shuah Khan wrote: > On 2/6/23 16:43, Luis Chamberlain wrote: > > The default sefltests timeout is 45 seconds. If you run the kmod > > selftests on your own with say: > > > > ./tools/testings/selftests/kmod.sh > > > > Then the default timeout won't be in effect. > > > > I've never ran kmod selftests using the generic make wrapper > > (./tools/testing/selftests/run_kselftest.sh -s) util now > > that I have support for it on kdevops [0]. And with that the > > test is limitted to the default timeout which we quickly run > > into. Bump this up to what I see is required on 8GiB / 8 vcpu > > libvirt q35 guest as can be easily created now with kdevops. > > > > To run selftests with kdevops: > > > > make menuconfig # enable dedicated selftests and kmod test > > make > > make bringup > > make linux > > make selftests-kmod > > > > This ends up taking about 280 seconds now, give or take add > > 50 seconds more more and we end up with 350. Document the > > rationale. > > > > [0] https://github.com/linux-kdevops/kdevops > > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> > > --- > > tools/testing/selftests/kmod/settings | 4 ++++ > > 1 file changed, 4 insertions(+) > > create mode 100644 tools/testing/selftests/kmod/settings > > > > diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings > > new file mode 100644 > > index 000000000000..6fca0f1a4594 > > --- /dev/null > > +++ b/tools/testing/selftests/kmod/settings > > @@ -0,0 +1,4 @@ > > +# measured from a manual run: > > +# time ./tools/testing/selftests/kmod/kmod.sh > > +# Then add ~50 seconds more gracetime. > > +timeout=350 > > Adding timeouts like this for individual tests increases the overall kselftest > run-time. I am not in favor of adding timeouts. > > We have to find a better way to do this. Well if folks don't have this the test will fail, and so a false positive. If the goal is to have a low time timeout for "do not run tests past this time and do not fail if we stopped the test" then that seems to be likely one way to go and each test may need to be modified to not fail fatally in case of a special signal. Luis ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 2023-02-27 22:42 ` Luis Chamberlain @ 2023-03-03 20:35 ` Shuah Khan 2023-03-03 21:48 ` Luis Chamberlain 0 siblings, 1 reply; 9+ messages in thread From: Shuah Khan @ 2023-03-03 20:35 UTC (permalink / raw) To: Luis Chamberlain Cc: shuah, linux-kselftest, gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel, Shuah Khan On 2/27/23 15:42, Luis Chamberlain wrote: > On Mon, Feb 27, 2023 at 03:32:50PM -0700, Shuah Khan wrote: >> On 2/6/23 16:43, Luis Chamberlain wrote: >>> The default sefltests timeout is 45 seconds. If you run the kmod >>> selftests on your own with say: >>> >>> ./tools/testings/selftests/kmod.sh >>> >>> Then the default timeout won't be in effect. >>> >>> I've never ran kmod selftests using the generic make wrapper >>> (./tools/testing/selftests/run_kselftest.sh -s) util now >>> that I have support for it on kdevops [0]. And with that the >>> test is limitted to the default timeout which we quickly run >>> into. Bump this up to what I see is required on 8GiB / 8 vcpu >>> libvirt q35 guest as can be easily created now with kdevops. >>> >>> To run selftests with kdevops: >>> >>> make menuconfig # enable dedicated selftests and kmod test >>> make >>> make bringup >>> make linux >>> make selftests-kmod >>> >>> This ends up taking about 280 seconds now, give or take add >>> 50 seconds more more and we end up with 350. Document the >>> rationale. >>> >>> [0] https://github.com/linux-kdevops/kdevops >>> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> >>> --- >>> tools/testing/selftests/kmod/settings | 4 ++++ >>> 1 file changed, 4 insertions(+) >>> create mode 100644 tools/testing/selftests/kmod/settings >>> >>> diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings >>> new file mode 100644 >>> index 000000000000..6fca0f1a4594 >>> --- /dev/null >>> +++ b/tools/testing/selftests/kmod/settings >>> @@ -0,0 +1,4 @@ >>> +# measured from a manual run: >>> +# time ./tools/testing/selftests/kmod/kmod.sh >>> +# Then add ~50 seconds more gracetime. >>> +timeout=350 >> >> Adding timeouts like this for individual tests increases the overall kselftest >> run-time. I am not in favor of adding timeouts. >> >> We have to find a better way to do this. > > Well if folks don't have this the test will fail, and so a false > positive. If the goal is to have a low time timeout for "do not run > tests past this time and do not fail if we stopped the test" then > that seems to be likely one way to go and each test may need to be > modified to not fail fatally in case of a special signal. > We are finding more and more that timeout values are requiring tweaks. I am in favor of coming up a way to exit the test with a timeout condition. thanks, -- Shuah ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 2023-03-03 20:35 ` Shuah Khan @ 2023-03-03 21:48 ` Luis Chamberlain 2023-03-06 16:06 ` Shuah Khan 0 siblings, 1 reply; 9+ messages in thread From: Luis Chamberlain @ 2023-03-03 21:48 UTC (permalink / raw) To: Shuah Khan Cc: shuah, linux-kselftest, gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel On Fri, Mar 03, 2023 at 01:35:10PM -0700, Shuah Khan wrote: > On 2/27/23 15:42, Luis Chamberlain wrote: > > On Mon, Feb 27, 2023 at 03:32:50PM -0700, Shuah Khan wrote: > > > On 2/6/23 16:43, Luis Chamberlain wrote: > > > > The default sefltests timeout is 45 seconds. If you run the kmod > > > > selftests on your own with say: > > > > > > > > ./tools/testings/selftests/kmod.sh > > > > > > > > Then the default timeout won't be in effect. > > > > > > > > I've never ran kmod selftests using the generic make wrapper > > > > (./tools/testing/selftests/run_kselftest.sh -s) util now > > > > that I have support for it on kdevops [0]. And with that the > > > > test is limitted to the default timeout which we quickly run > > > > into. Bump this up to what I see is required on 8GiB / 8 vcpu > > > > libvirt q35 guest as can be easily created now with kdevops. > > > > > > > > To run selftests with kdevops: > > > > > > > > make menuconfig # enable dedicated selftests and kmod test > > > > make > > > > make bringup > > > > make linux > > > > make selftests-kmod > > > > > > > > This ends up taking about 280 seconds now, give or take add > > > > 50 seconds more more and we end up with 350. Document the > > > > rationale. > > > > > > > > [0] https://github.com/linux-kdevops/kdevops > > > > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> > > > > --- > > > > tools/testing/selftests/kmod/settings | 4 ++++ > > > > 1 file changed, 4 insertions(+) > > > > create mode 100644 tools/testing/selftests/kmod/settings > > > > > > > > diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings > > > > new file mode 100644 > > > > index 000000000000..6fca0f1a4594 > > > > --- /dev/null > > > > +++ b/tools/testing/selftests/kmod/settings > > > > @@ -0,0 +1,4 @@ > > > > +# measured from a manual run: > > > > +# time ./tools/testing/selftests/kmod/kmod.sh > > > > +# Then add ~50 seconds more gracetime. > > > > +timeout=350 > > > > > > Adding timeouts like this for individual tests increases the overall kselftest > > > run-time. I am not in favor of adding timeouts. > > > > > > We have to find a better way to do this. > > > > Well if folks don't have this the test will fail, and so a false > > positive. If the goal is to have a low time timeout for "do not run > > tests past this time and do not fail if we stopped the test" then > > that seems to be likely one way to go and each test may need to be > > modified to not fail fatally in case of a special signal. > > > > We are finding more and more that timeout values are requiring > tweaks. I am in favor of coming up a way to exit the test with > a timeout condition. OK so do we use the existing timeout as a "optional, I don't want my test to take longer than this" or "if this test takes longer than this amount this is a fatal issue"? I ask because right now we can't override it even with an environment variable. If we had such support we can let test runners (like kdevops) use selftests with its own set of qualified / verified timeouts for the VMs it uses. For instance, Iw ant to soon start asking 0day to enable my kdevops 0-day tests for the subsystems I maintain, but I can't do that yet as the timeout is not correct. Luis ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 2023-03-03 21:48 ` Luis Chamberlain @ 2023-03-06 16:06 ` Shuah Khan 2023-03-08 20:29 ` Luis Chamberlain 0 siblings, 1 reply; 9+ messages in thread From: Shuah Khan @ 2023-03-06 16:06 UTC (permalink / raw) To: Luis Chamberlain Cc: shuah, linux-kselftest, gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel, Shuah Khan On 3/3/23 14:48, Luis Chamberlain wrote: > On Fri, Mar 03, 2023 at 01:35:10PM -0700, Shuah Khan wrote: >> On 2/27/23 15:42, Luis Chamberlain wrote: >>> On Mon, Feb 27, 2023 at 03:32:50PM -0700, Shuah Khan wrote: >>>> On 2/6/23 16:43, Luis Chamberlain wrote: >>>>> The default sefltests timeout is 45 seconds. If you run the kmod >>>>> selftests on your own with say: >>>>> >>>>> ./tools/testings/selftests/kmod.sh >>>>> >>>>> Then the default timeout won't be in effect. >>>>> >>>>> I've never ran kmod selftests using the generic make wrapper >>>>> (./tools/testing/selftests/run_kselftest.sh -s) util now >>>>> that I have support for it on kdevops [0]. And with that the >>>>> test is limitted to the default timeout which we quickly run >>>>> into. Bump this up to what I see is required on 8GiB / 8 vcpu >>>>> libvirt q35 guest as can be easily created now with kdevops. >>>>> >>>>> To run selftests with kdevops: >>>>> >>>>> make menuconfig # enable dedicated selftests and kmod test >>>>> make >>>>> make bringup >>>>> make linux >>>>> make selftests-kmod >>>>> >>>>> This ends up taking about 280 seconds now, give or take add >>>>> 50 seconds more more and we end up with 350. Document the >>>>> rationale. >>>>> >>>>> [0] https://github.com/linux-kdevops/kdevops >>>>> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> >>>>> --- >>>>> tools/testing/selftests/kmod/settings | 4 ++++ >>>>> 1 file changed, 4 insertions(+) >>>>> create mode 100644 tools/testing/selftests/kmod/settings >>>>> >>>>> diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings >>>>> new file mode 100644 >>>>> index 000000000000..6fca0f1a4594 >>>>> --- /dev/null >>>>> +++ b/tools/testing/selftests/kmod/settings >>>>> @@ -0,0 +1,4 @@ >>>>> +# measured from a manual run: >>>>> +# time ./tools/testing/selftests/kmod/kmod.sh >>>>> +# Then add ~50 seconds more gracetime. >>>>> +timeout=350 >>>> >>>> Adding timeouts like this for individual tests increases the overall kselftest >>>> run-time. I am not in favor of adding timeouts. >>>> >>>> We have to find a better way to do this. >>> >>> Well if folks don't have this the test will fail, and so a false >>> positive. If the goal is to have a low time timeout for "do not run >>> tests past this time and do not fail if we stopped the test" then >>> that seems to be likely one way to go and each test may need to be >>> modified to not fail fatally in case of a special signal. >>> >> >> We are finding more and more that timeout values are requiring >> tweaks. I am in favor of coming up a way to exit the test with >> a timeout condition. > > OK so do we use the existing timeout as a "optional, I don't want my > test to take longer than this" or "if this test takes longer than > this amount this is a fatal issue"? It isn't a fatal issue. So I wouldn't call it one. I would add a message saying test timed out. One way to handle this is: - Add a test run-time option and have user tune it as needed. Make the timeout an option so users can set it based on their environments. > > I ask because right now we can't override it even with an environment > variable. If we had such support we can let test runners (like kdevops) > use selftests with its own set of qualified / verified timeouts for the > VMs it uses. > > For instance, Iw ant to soon start asking 0day to enable my kdevops > 0-day tests for the subsystems I maintain, but I can't do that yet as > the timeout is not correct. This test isn't part of the default run, so day has to run this as a special case and it would make prefect sense to provide a tunable timeout option. thanks, -- Shuah > > Luis ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 2023-03-06 16:06 ` Shuah Khan @ 2023-03-08 20:29 ` Luis Chamberlain 0 siblings, 0 replies; 9+ messages in thread From: Luis Chamberlain @ 2023-03-08 20:29 UTC (permalink / raw) To: Shuah Khan Cc: shuah, linux-kselftest, gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel, Vincenzo Palazzo, Lucas De Marchi On Mon, Mar 06, 2023 at 09:06:41AM -0700, Shuah Khan wrote: > On 3/3/23 14:48, Luis Chamberlain wrote: > > On Fri, Mar 03, 2023 at 01:35:10PM -0700, Shuah Khan wrote: > > > On 2/27/23 15:42, Luis Chamberlain wrote: > > > > On Mon, Feb 27, 2023 at 03:32:50PM -0700, Shuah Khan wrote: > > > > > On 2/6/23 16:43, Luis Chamberlain wrote: > > > > > > The default sefltests timeout is 45 seconds. If you run the kmod > > > > > > selftests on your own with say: > > > > > > > > > > > > ./tools/testings/selftests/kmod.sh > > > > > > > > > > > > Then the default timeout won't be in effect. > > > > > > > > > > > > I've never ran kmod selftests using the generic make wrapper > > > > > > (./tools/testing/selftests/run_kselftest.sh -s) util now > > > > > > that I have support for it on kdevops [0]. And with that the > > > > > > test is limitted to the default timeout which we quickly run > > > > > > into. Bump this up to what I see is required on 8GiB / 8 vcpu > > > > > > libvirt q35 guest as can be easily created now with kdevops. > > > > > > > > > > > > To run selftests with kdevops: > > > > > > > > > > > > make menuconfig # enable dedicated selftests and kmod test > > > > > > make > > > > > > make bringup > > > > > > make linux > > > > > > make selftests-kmod > > > > > > > > > > > > This ends up taking about 280 seconds now, give or take add > > > > > > 50 seconds more more and we end up with 350. Document the > > > > > > rationale. > > > > > > > > > > > > [0] https://github.com/linux-kdevops/kdevops > > > > > > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> > > > > > > --- > > > > > > tools/testing/selftests/kmod/settings | 4 ++++ > > > > > > 1 file changed, 4 insertions(+) > > > > > > create mode 100644 tools/testing/selftests/kmod/settings > > > > > > > > > > > > diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings > > > > > > new file mode 100644 > > > > > > index 000000000000..6fca0f1a4594 > > > > > > --- /dev/null > > > > > > +++ b/tools/testing/selftests/kmod/settings > > > > > > @@ -0,0 +1,4 @@ > > > > > > +# measured from a manual run: > > > > > > +# time ./tools/testing/selftests/kmod/kmod.sh > > > > > > +# Then add ~50 seconds more gracetime. > > > > > > +timeout=350 > > > > > > > > > > Adding timeouts like this for individual tests increases the overall kselftest > > > > > run-time. I am not in favor of adding timeouts. > > > > > > > > > > We have to find a better way to do this. > > > > > > > > Well if folks don't have this the test will fail, and so a false > > > > positive. If the goal is to have a low time timeout for "do not run > > > > tests past this time and do not fail if we stopped the test" then > > > > that seems to be likely one way to go and each test may need to be > > > > modified to not fail fatally in case of a special signal. > > > > > > > > > > We are finding more and more that timeout values are requiring > > > tweaks. I am in favor of coming up a way to exit the test with > > > a timeout condition. > > > > OK so do we use the existing timeout as a "optional, I don't want my > > test to take longer than this" or "if this test takes longer than > > this amount this is a fatal issue"? > > It isn't a fatal issue. So I wouldn't call it one. I would add a > message saying test timed out. > > One way to handle this is: > - Add a test run-time option and have user tune it as needed. > > Make the timeout an option so users can set it based on their > environments. > > > > > I ask because right now we can't override it even with an environment > > variable. If we had such support we can let test runners (like kdevops) > > use selftests with its own set of qualified / verified timeouts for the > > VMs it uses. > > > > For instance, Iw ant to soon start asking 0day to enable my kdevops > > 0-day tests for the subsystems I maintain, but I can't do that yet as > > the timeout is not correct. > > This test isn't part of the default run, so day has to run this as a > special case and it would make prefect sense to provide a tunable > timeout option. That's the thing, I *want* it to be part of *my runs* for my git trees on git.kernel.org for modules-testing and modules-next. That allows me to have 0day run whatever things I need. Long term it will be through kdevops with just: make linux make selftests-kmod make kmod make kmod-check Vincenzo expressed interest to help, I think he may be interested in helping with this configurable timeout for selftests as some initial low hanging fruit. Then on the kdevops front we'd set what we know is right for a typical libvirt use case for our current default VM target. Luis ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/2] selftests/firmware: increase timeout from 165 to 230 2023-02-06 23:43 [PATCH 0/2] selftests: bump timeout for firmware and kmod Luis Chamberlain 2023-02-06 23:43 ` [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 Luis Chamberlain @ 2023-02-06 23:43 ` Luis Chamberlain 1 sibling, 0 replies; 9+ messages in thread From: Luis Chamberlain @ 2023-02-06 23:43 UTC (permalink / raw) To: shuah, linux-kselftest Cc: gregkh, tiwai, tianfei.zhang, russell.h.weight, keescook, tweek, a.manzanares, dave, linux-modules, linux-kernel, Luis Chamberlain Bump the timeout up to what we can empirally need on a q35 8MiB system / 8 vcpus as supported easily now with kdevops [0]. To test firmware_loader with kdevops: make menuconfig # enable dedicated selftests and firmware test make make bringup make linux make selftests-firmware In practice this test now takes about 170 seconds so let's give it a bit more breathing room we end up with 230. Note: I'm seeing a failure only on Android setups running this test in a loop, we eventually OOM on linux-next tag next-20230119. [0] https://github.com/linux-kdevops/kdevops Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> --- tools/testing/selftests/firmware/settings | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/firmware/settings b/tools/testing/selftests/firmware/settings index 085e664ee093..75773074af35 100644 --- a/tools/testing/selftests/firmware/settings +++ b/tools/testing/selftests/firmware/settings @@ -5,4 +5,10 @@ # Additionally, fw_fallback may take 5 seconds for internal timeouts in each # of the 3 configs, so at least another 15 seconds are needed. Add another # 10 seconds for each testing config: 120 + 15 + 30 -timeout=165 +# +# That's 165.. but we also now have some other batched tests, we get the +# current timeout value by running manually withtout the timeout: +# +# time ./tools/testing/selftests/firmware/fw_run_tests.sh +# then add give or take about 50 seconds. +timeout=230 -- 2.39.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2023-03-08 20:30 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-02-06 23:43 [PATCH 0/2] selftests: bump timeout for firmware and kmod Luis Chamberlain 2023-02-06 23:43 ` [PATCH 1/2] selftests/kmod: increase the kmod timeout from 45 to 165 Luis Chamberlain 2023-02-27 22:32 ` Shuah Khan 2023-02-27 22:42 ` Luis Chamberlain 2023-03-03 20:35 ` Shuah Khan 2023-03-03 21:48 ` Luis Chamberlain 2023-03-06 16:06 ` Shuah Khan 2023-03-08 20:29 ` Luis Chamberlain 2023-02-06 23:43 ` [PATCH 2/2] selftests/firmware: increase timeout from 165 to 230 Luis Chamberlain
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.