[U-Boot] [PATCH] Implement pytest-based test infrastructure

From: Stephen Warren <swarren@wwwdotorg.org>
To: u-boot@lists.denx.de
Subject: [U-Boot] [PATCH] Implement pytest-based test infrastructure
Date: Tue, 24 Nov 2015 14:28:27 -0700	[thread overview]
Message-ID: <5654D67B.2090307@wwwdotorg.org> (raw)
In-Reply-To: <CAPnjgZ1EJPH6P2b870auwgQkDW53TyFZhp_BrWN5oRDjnZ-cAw@mail.gmail.com>

On 11/24/2015 12:04 PM, Simon Glass wrote:
> Hi Stephen,
>
> On 23 November 2015 at 21:44, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> On 11/23/2015 06:45 PM, Simon Glass wrote:
>>> On 22 November 2015 at 10:30, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>> On 11/21/2015 09:49 AM, Simon Glass wrote:

>>>>> OK I got it working thank you. It is horribly slow though - do you
>>>>> know what is holding it up? For me to takes 12 seconds to run the
>>>>> (very basic) tests.
..
>> I put a bit of time measurement into run_command() and found that on my
>> system at work, for p.send("the shell command to execute") was actually
>> (marginally) slower on sandbox than on real HW, despite real HW being a
>> 115200 baud serial port, and the code splitting the shell commands into
>> chunks that are sent and waited for synchronously to avoid overflowing
>> UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it
>> seems to be non-blocking, so I don't think termios VMIN/VTIME come into
>> play (setting them to 0 made no difference), and the two raw modes took
>> the same time. I meant to look into pexpect's termios settings to see if
>> there was anything to tweak there, but forgot today.
>>
>> I did do one experiment to compare expect (the Tcl version) and pexpect.
>> If I do roughly the following in both:
>>
>> spawn u-boot (sandbox)
>> wait for prompt
>> 100 times:
>>      send "echo $foo\n"
>>      wait for "echo $foo"
>>      wait for shell prompt
>> send "reset"
>> wait for "reset"
>> send "\n"
>>
>> ... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I
>> remove all the "wait"s, then IIRC Tcl was about 15x faster or more.
>> That's a pity. Still, I'm sure as heck not going to rewrite all this in
>> Tcl:-( I wonder if something similar to pexpect but more targetted at
>> simple "interactive shell" cases would remove any of that overhead.
>
> It is possible that we should use sandbox in 'cooked' mode so that
> lines an entered synchronously. The -t option might help here, or we
> may need something else.

I don't think cooked mode will work, since I believe cooked is 
line-buffered, yet when U-Boot emits the shell prompt there's no \n 
printed afterwards.

FWIW, I hacked out pexpect and replaced it with some custom code. That 
reduced by sandbox execution time from ~5.1s to ~2.3s. Execution time 
against real HW didn't seem to be affected at all. Some features like 
timeouts and complete error handling are still missing, but I don't 
think that would affect the execution time. See my github tree for the 
WIP patch.