Re: [PATCH 2/2] ftrace/selftest: absence of modules/programs should trigger unsupported errors

[Alan Maguire <alan.maguire@oracle.com>]

On Fri, 7 Feb 2020, Masami Hiramatsu wrote:

> On Fri, 7 Feb 2020 08:27:13 +0000 (GMT)
> Alan Maguire <alan.maguire@oracle.com> wrote:
> 
> > On Fri, 7 Feb 2020, Masami Hiramatsu wrote:
> > 
> > > Hi Alan,
> > > 
> > > On Thu,  6 Feb 2020 15:09:20 +0000
> > > Alan Maguire <alan.maguire@oracle.com> wrote:
> > > 
> > > > In a number of cases, the ftrace tests check for the presence of
> > > > ftrace testing-related modules (ftrace-direct, trace-printk) and
> > > > programs (checkbashisms), returning exit_unresolved if these
> > > > are not found.  The problem is, exit_unresolved causes execution
> > > > of ftracetest to return an error, when really our tests are
> > > > failing due to not having the requisite kernel configuration/tools
> > > > present, which is I think more of an unsupported error condition.
> > > > With these fixed, we see no unresolved test cases and ftracetest
> > > > returns success ("ok" when run via kselftest).
> > > 
> > > If your problem is to pass the test even if you don't test the
> > > feature, please change the ftracetest itself instead of replacing
> > > unresolved with unsupported. Those notice different situation.
> > > 
> > > unresolved - Testcase can not find some tools or helper drivers
> > >              which are required for this testcase.
> > > 
> > > unsupported - Kernel does not have tested feature because of
> > >               the version or the configuration.
> > > 
> > > Obviously the unresolved is a test environment issue. No test-module
> > > doesn't mean no feature to be tested.
> > > Could you tell me the reason why you can't install those required
> > > tools and modules on the test environment?
> > > 
> > 
> > Sure! In my case, I'm testing a distro production kernel,
> > where I can't control the CONFIG variable settings.  In
> > this case, ideally I'd like the tests to return success
> > if no problems with ftrace were detected, even if some
> > of the tests could not be run due to missing modules
> > and programs.
> 
> OK, for modules, we need to find another way to solve the issue.
> But how about checkbashisms? you can download and build it.
> 
> https://sources.debian.org/src/devscripts/2.20.2/
>

Yep, I should have said that this one isn't a big issue, it's
also packaged in some distros in rpmdevtools.
 
> For the modules, you might be able to build it from kernel
> source code as out-of-tree modules, or not?
> (hmm, how do the other test handle it...?)
>

Ideally (from my perspective at least) the tests would
build the modules if needed, but I think the constraints
on kselftests are  that sometimes the kselftests are packaged
and  installed without the rest of the source tree, so I
_think_ given that the module source is in other parts of the
tree (that may not be present) we're probably stuck.  It'd be
great to have have a solution to this though, as I have a
feeling there may be more and more cases like this with the
growth of kunit.

Looks like the official way to do this sort of thing is
described in Documentation/dev-tools/kselftest.rst:

   # Assumes you have booted a fresh build of this kernel tree
   cd /path/to/linux/tree
   make kselftest-merge
   make modules
   sudo make modules_install
   make TARGETS=lib kselftest

It seems to merge existing config with the config files in the
kselftest dirs and rebuild modules; I'll give this a try.

I was hoping for a lightweight version of the above which
just builds the modules needed without rebuilding everything.

> >  As you suggest above (unless I'm
> > misunderstanding), this could be accomplished by modifying
> > ftracetest itself.  Would doing something like what is done
> > for UNSUPPORTED_RESULT (defaults to 0, but can be set to
> > 1 via --fail-unsupported, such that ftracetest returns
> > 1 if we encounter unsupported results) make sense for
> > the unresolved case too?
> 
> Yes, but at first could you try to setup your testing environment?
> If you are officially testing your distro kernel, the distro
> might need to be tested with full-set of testcases.
>

Absolutely! I'll see if I can convince the modules to
build using the above scheme.

Thanks for the review and advice!

Alan

> If not (like you are testing kernel for fun :)), you can just
> make your custom set of testcases. (just remove those test files)
> 
> Thank you,
> 
> 
> -- 
> Masami Hiramatsu <mhiramat@kernel.org>
>