From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stephen Warren Date: Tue, 5 Dec 2017 11:20:57 -0700 Subject: [U-Boot] [RFC PATCH] test: py: Disable sleep test for qemu targets In-Reply-To: <20171204232107.GL3587@bill-the-cat> References: <55ea4a9704460aeefaf48b17ed7ab19c8b0285fc.1512139562.git.michal.simek@xilinx.com> <346f23bc-7e7b-6fcb-5f53-8ad674febd60@gmx.de> <9babee0f-f5a8-d66a-db6f-ead7c4ac7dab@xilinx.com> <20171201224408.GV3587@bill-the-cat> <20171204140314.GB3587@bill-the-cat> <20171204153006.GF3587@bill-the-cat> <13e38b70-c2c6-1129-6ef3-9ed5366fe5ea@wwwdotorg.org> <20171204232107.GL3587@bill-the-cat> Message-ID: <08810498-f865-9930-27a1-95e1340f68aa@wwwdotorg.org> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: u-boot@lists.denx.de On 12/04/2017 04:21 PM, Tom Rini wrote: > On Mon, Dec 04, 2017 at 10:14:06AM -0700, Stephen Warren wrote: >> On 12/04/2017 08:30 AM, Tom Rini wrote: >>> On Mon, Dec 04, 2017 at 03:21:04PM +0100, Michal Simek wrote: >>>> On 4.12.2017 15:03, Tom Rini wrote: >>>>> On Mon, Dec 04, 2017 at 02:55:45PM +0100, Michal Simek wrote: >>>>>> On 1.12.2017 23:44, Tom Rini wrote: >>>>>>> On Fri, Dec 01, 2017 at 10:07:54AM -0700, Stephen Warren wrote: >>>>>>>> On 12/01/2017 08:19 AM, Michal Simek wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> On 1.12.2017 16:06, Heinrich Schuchardt wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 12/01/2017 03:46 PM, Michal Simek wrote: >>>>>>>>>>> Qemu for arm32/arm64 has a problem with time setup. >>>>>>>>>> >>>>>>>>>> Wouldn't it be preferable to fix the root cause? >>>>>>>>> >>>>>>>>> Definitely that would be the best and IIRC I have tried to convince our >>>>>>>>> qemu guy to do that but they have never done that. >>>>>>>> >>>>>>>> What is the exact failure condition? Is it simply that the test is still >>>>>>>> slightly too strict about which delays it accepts, or is sleep outright >>>>>>>> broken? >>>>>>>> >>>>>>>> You can use command-line option -k to avoid some tests. For example "-k not >>>>>>>> sleep". That way, we don't have to hard-code the dependency into the test >>>>>>>> source. Depending on the root cause (issue in U-Boot, or issue in just your >>>>>>>> local version of qemu, or something that will never work) this might be >>>>>>>> better? >>>>>>> >>>>>>> Even with the most recent relaxing of the sleep test requirements, I can >>>>>>> still (depending on overall system load) have 'sleep' take too long, on >>>>>>> QEMU. I think it might have been half a second slow, but I don't have >>>>>>> the log handy anymore. Both locally and in travis we -k not sleep all >>>>>>> of the qemu instances. >>>>>> >>>>>> ok. By locally do you mean just using -k not sleep? >>>>> >>>>> Yes, I have that in my CI scripts and similar. >>>> >>>> Wouldn't be easier to keep this in uboot-test-hooks repo with other >>>> target setting? >>> >>> Or do as you did did and mark the tests as not allowed for qemu, yes. >>> >>>> What we are trying to do is that our testing group will run these tests >>>> for me that's why it is just easier for me to change local >>>> uboot-test-hooks repo instead of communicate with them what -k not XXX >>>> parameters to add to certain scripts. >>>> >>>> It means in loop they will just run all tests on qemu, local targets and >>>> in boardfarm. It is probably not big deal to tell them to add -k not >>>> sleep for all qemu runs but I know that for some i2c testing qemu >>>> doesn't emulate these devices that's why these tests fails. And the >>>> amount of tests which we shouldn't run on qemu will probably grow. >>> >>> Well, I'm still open to possibly tweaking the allowed variance in the >>> sleep test. OTOH, if we just say "no QEMU" here, we can then go back to >>> "sleep should be pretty darn accurate on HW" for the test too, and >>> perhaps that's best. >> >> The fundamental problem of "over-sleeping" due to host system load/.. exists >> with all boards. There's nothing specific to qemu here except that running >> U-Boot on qemu on the host rather than on separate HW might more easily >> trigger the "high load on the host" condition; I see the issue now and then >> and manually retry that test, although that is a bit annoying. >> >> The original test was mostly intended to make sure that e U-Boot clock >> didn't run at a significantly different rate to the host, since I had seen >> that issue during development of some board support or as a regression >> sometime. Perhaps the definition of "significantly different" should be more >> like "1/2 rate or twice rate or more" rather than "off by a small fraction >> of a second". That might avoid so many false positives. > > I've pushed this up to 10 seconds and 0.5s worth of overrun and on > qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast > and to me a wrong-clocking value, yes? For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5 seconds (including my reaction time), so ~13.2 does sound like it's probably a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with real-time? I'd hope not for something simple like this, assuming you're using a recent CPU, but maybe.