[U-Boot] [PATCH 00/23] efi_loader implement missing functions

[Leif Lindholm <leif.lindholm@linaro.org>]

On Wed, Aug 30, 2017 at 12:03:16AM +0200, Heinrich Schuchardt wrote:
> On 08/29/2017 10:38 PM, Alexander Graf wrote:
> >> Am 29.08.2017 um 22:16 schrieb Simon Glass <sjg@chromium.org>:
> >>>>>> That seems more useful long term than re-inventing comprehensive UEFI
> >>>>>> test suite.  (Also, beyond just running shim/fallback/grub, I don't
> >>>>>> really have time to invent new tests for the stack of efi_loader
> >>>>>> patches I have.)
> >>>>>
> >>>>> Yes and no - it depends on the availability of the official suite :/.
> >>>>
> >>>> UEFI SCT is not yet open source/free software. Obviously, this is
> >>>> something Linaro has been lobbying for since day one of our
> >>>> involvement. There used to be little understanding for this, but that
> >>>> attitude has shifted substantially.
> >>>
> >>> So, if/until UEFI SCT is not an option, what about:
> >>>
> >>>  https://01.org/linux-uefi-validation
> >>>
> >>> (thx to pjones for pointing that out to me)
> >>
> >> Well in any case I'm not looking for a large functional test suite at
> >> this stage. It certainly could be useful, but not as a replacement for
> >> unit tests. The latter is for fast verification (so that everyone can
> >> run it as part of 'make tests') and easy identification of the
> >> location of bugs.
> > 
> > I fail to see the point here to be honest. The thing we care most
> > about in the uefi implementation is abi compatibility with the
> > spec. That's basically what SCT is supposed to get you. Running it
> > should be a matter of seconds with kvm enabled, so 'make tests'
> > should easily be able to run it if you have the correct target
> > compiled.
> > 
> >>
> >> These new tests should be written in C, run very quickly (similar to
> >> the existing tests) to verify that the code works. Then when each new
> >> feature is added, the test are expanded to cover the new
> >> functionality. Should the code be refactored, the tests should provide
> >> comfort that nothing broke.
> > 
> > So far I've never seen that approach work well in (low level) open
> > source projects, as you start to heavily raise the barrier for
> > participation. Most people in this world do these tasks in their
> > spare time and won't spend x2 the time to also write and update
> > unit tests.
> > 
> > I really think having a working functional test suite that is easy
> > to run is the best way forward in this case. We can then start
> > thinking of ways to test hard to reach code sections to increase
> > test coverage.
> > 
> > Alex
> 
> Simon is right in that test code is needed to ensure that refactoring
> does not break existing functionality. The current git HEAD cannot read
> from an IDE disk anymore. This problem could have been caught by a
> functional test four weeks ago.
> 
> Unit tests should be based on exposed APIs. Tests for lower level
> functions would not survive refactoring.

Completely agreed. UEFI is an interface, not an implementation.

> We implement only part of the UEFI spec. If we do not want to blow up
> binary size we may even want to keep it that way. So a test suite for
> full UEFI compatibility will always fail partially if it runs at all.

SCT is (almost?) entirely implemented in the form of:
- Does the implementation claim to support X?
- If so, determine if X behaves in conformance with the specification.

> I am able to provide a test application that will cover the API
> functions that I have focused on (events, protocols, (dis)connect
> controllers). To limit my effort I would like to write it for the HEAD
> of my patch queue and not break it down into one test patch per
> implementation patch. I hope that is ok for you. My effort estimate is
> 40h+ so, please, be patient.

I am not going to claim that getting SCT to run under U-Boot is going
to come near the top of my priority queue any time soon, and it would
certainly be useful for some sort of "make sure we don't break what we
have" tests.

But there will be additional bits of UEFI functionality that will be
useful to add. Some of that functionality will be generically useful,
some may not be relevant for certain systems and if the effect on code
size is non-negligable, its inclusion in the final image should be
made conditional.

And surely it would be preferable to be able to reuse an existing
testsuite when adding new bits, instead of having to implement new
tests for every feature? Especially when the goal is compatibility
with something that uses that existing test suite.

Alex got me up and running with a test environment, so I will put
"investigate why Shell.efi won't run" on my background task list.

/
    Leif