Re: travis: any reason we have keep going on make commands

From: Nicolas Iooss <nicolas.iooss@m4x.org>
To: Paul Moore <paul@paul-moore.com>,
	William Roberts <bill.c.roberts@gmail.com>
Cc: SElinux list <selinux@vger.kernel.org>,
	Stephen Smalley <stephen.smalley.work@gmail.com>
Subject: Re: travis: any reason we have keep going on make commands
Date: Fri, 10 Jul 2020 11:08:28 +0200	[thread overview]
Message-ID: <CAJfZ7=kyieBs6_NPE3k2tyNSydrwmVT7k3kcmd=0Xsi7JoD2Zg@mail.gmail.com> (raw)
In-Reply-To: <CAHC9VhQkTgYFQ5ncLDJpOcqynWam3QY7udLQoUYSb3HgkvdboQ@mail.gmail.com>

On Thu, Jul 9, 2020 at 8:00 PM Paul Moore <paul@paul-moore.com> wrote:
>
> On Thu, Jul 9, 2020 at 10:33 AM William Roberts
> <bill.c.roberts@gmail.com> wrote:
> > So Nicolas initially created our travis script in commit c9adfe2d2653
> > and has -k, or keep going, on the make commands. This causes make to
> > plow ahead and bury the errors in the logs. Stephen noticed this the
> > other day, and we have been chatting about it out of band and wanted
> > to pull in the community.
> >
> > Are their compelling reasons for keeping this behavior? I am also
> > concerned that we could get false positives on travis success results.
>
> In my opinion the whole point of automated testing is to catch
> failures early and often.  For that reason I would want the test to
> fail and stop, both because I find it easier to identify the failure
> that way and also because I'm not sure I would trust much of the
> testing that occurred after an error condition.
>
Hi,
There seems to be some confusion:

* "make -k" does not stop the "make" command at the first error and
allows seeing all the errors when there are several ones. In my humble
opinion, it makes sense when compiling ("make all") and not when
running tests ("make test"), and this is actually what is right now in
Travis-CI. "make -k" returns a failure exit code when an error
happens.

* Travis-CI does not stop the job as soon as a sub-command fails. If I
understand correctly, this is what really bothers William, and I agree
this is a behavior that can be improved. According to
https://github.com/travis-ci/travis-ci/issues/1066, a possible
solution could be to use "set -e", which could have unexpected
side-effects in launched commands. It is possible to "emulate set -e"
by adding exit statements, such as :

    - make install $EXPLICIT_MAKE_VARS -k || exit $?
    - make install-pywrap $EXPLICIT_MAKE_VARS -k || exit $?
    - make install-rubywrap $EXPLICIT_MAKE_VARS -k || exit $?
    # ...
    - make test $EXPLICIT_MAKE_VARS || exit $?

I have not tested whether this works on Travis-CI, but if it does, it
would be a nice improvement. I will take a look this week-end.

Cheers,
Nicolas