From mboxrd@z Thu Jan  1 00:00:00 1970
From: Simon Glass <sjg@chromium.org>
Date: Mon, 21 Oct 2019 16:53:59 -0600
Subject: [U-Boot] [PATCH] test/py: hush_if_test: Add tests to cover
	octal/hex values
In-Reply-To: <2bc39daf-b1e2-0996-2d41-ca53de9fbb70@xilinx.com>
References: <2861e9ee042ad93a9d36f551cd90ce7cbc6030aa.1570707876.git.michal.simek@xilinx.com>
 <CAPnjgZ0kAj_shhbJ-=RP1Q6HRCWxCnyXMBt7-BFSbK-MYT7_4A@mail.gmail.com>
 <54db2685-3437-eed8-73fa-63d95a59b3ea@xilinx.com>
 <CAPnjgZ3uuXuH6RUUt4hbUKpCiYYHLfKuYB=GvteKhb+3j8HCtg@mail.gmail.com>
 <2bc39daf-b1e2-0996-2d41-ca53de9fbb70@xilinx.com>
Message-ID: <CAPnjgZ0CJuUZOSnvHMAoRrWA9ihKK5=Gg4MFjN+2Tqm=LGuHcg@mail.gmail.com>
List-Id: <u-boot.lists.denx.de>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
To: u-boot@lists.denx.de

Hi Michal,

On Tue, 15 Oct 2019 at 00:09, Michal Simek <michal.simek@xilinx.com> wrote:
>
> Hi Simon,
>
> On 11. 10. 19 17:53, Simon Glass wrote:
> > Hi Michal,
> >
> > On Fri, 11 Oct 2019 at 01:50, Michal Simek <michal.simek@xilinx.com> wrote:
> >>
> >> On 10. 10. 19 19:06, Simon Glass wrote:
> >>> Hi Michal,
> >>>
> >>> On Thu, 10 Oct 2019 at 05:44, Michal Simek <michal.simek@xilinx.com> wrote:
> >>>>
> >>>> Extend test suite to cover also automatic octal/hex converstions which
> >>>> haven't been implemented in past.
> >>>>
> >>>> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> >>>> ---
> >>>>
> >>>> Depends on https://lists.denx.de/pipermail/u-boot/2019-September/383309.html
> >>>>
> >>>> There are of course other tests which we can run but not sure if make sense
> >>>> to have there all combinations. The most interesting are mixed tests which
> >>>> are failing before patch above is applied.
> >>>> Definitely please let me know if you want to add any other test.
> >>>> ---
> >>>>  test/py/tests/test_hush_if_test.py | 27 +++++++++++++++++++++++++++
> >>>>  1 file changed, 27 insertions(+)
> >>>>
> >>>
> >>> I worry that these tests might be very slow since it requires a lot of
> >>> interaction with U-Boot over a pipe. Is it possible to put them in C
> >>> code instead, e.g. cmd_ut?
> >>
> >> I have of course running it on my HW and it is quite fast. It is just 16
> >> more simple tests. And if this breaks gitlab/travis CI loops then we
> >> have bigger problem.
> >
> > I mean running these tests on sandbox. The interactions with the
> > sandbox command line are quite slow I think.
>
>
> I am not sharing this concern.
>
> Before:
> [u-boot]$ time ./test/py/test.py --bd sandbox -s -k hush >/dev/null
>
> real    0m2,403s
> user    0m1,263s
> sys     0m0,299s
>
> After
> [u-boot]$ time ./test/py/test.py --bd sandbox -s -k hush >/dev/null
>
> real    0m2,864s
> user    0m1,563s
> sys     0m0,305s
>
> And if 0.4s on testing will cause issues somewhere else we have
> different kind of problem.

+Stephen Warren

I originally mentioned this concern to Stephen we the test setup was
created. At present even 'make qcheck' takes over a minute. Adding
half a second to this every time we add a new test is not going to
lead to a good place.

Stephen made some improvements to speed things up, and suggested that
the problem would not bear out. The alternative was presumably to
build U-Boot into a Python module to avoid the comms overhead. But we
didn't go that path.

So I think we should only use Python when the tests cannot be written in C.

Regards,
Simon