Multiple compiler support in the backend

Multiple compiler support in the backend

[Matt Hart]

Hi all,

Just wanted to clear up what my kernelci-backend PR [1] is for.

It introduces a build-environment variable to the build document,
which will allow the backend to distinguish between builds of the same
defconfig, but in a different environment. This variable is required
and requests without it will be dropped, to stop builds from
overwriting each other on the storage filesystem.

For kernelci.org usage, the environment difference will the compiler,
allowing our builders to push builds from our different docker images,
but as it's just a string, I'm hoping that the name build-environment
will work for other use cases.
There is already a compiler field which is populated by the backend
from the build.json and some simple parsing.

There is a matching PR [2] to kernelci-core-staging to add a
build-environment string to our builds (example:
docker-kernelci/build-gcc-7_arm64) but it has been suggested this
could be shortened to 'gcc-7' which I'm happy to change.

For the frontend, this change "just works", because build.py is now
populating the file_server_resource field, all the URLs on the build
information pages will be correct.

I would really like to get this deployed to staging for testing and
then rolled out ASAP as it's a requirement for starting builds with
compilers other than GCC, and multiple versions of GCC.


[1] https://github.com/kernelci/kernelci-backend/pull/86
[2] https://github.com/kernelci/kernelci-core-staging/pull/43

Re: [kernelci] Multiple compiler support in the backend

[Guillaume Tucker]

[-- Attachment #1: Type: text/plain, Size: 2596 bytes --]

On Fri, Nov 16, 2018 at 12:52 PM Matt Hart <matthew.hart@linaro.org> wrote:

> Hi all,
>
> Just wanted to clear up what my kernelci-backend PR [1] is for.
>
> It introduces a build-environment variable to the build document,
> which will allow the backend to distinguish between builds of the same
> defconfig, but in a different environment. This variable is required
> and requests without it will be dropped, to stop builds from
> overwriting each other on the storage filesystem.
>
> For kernelci.org usage, the environment difference will the compiler,
> allowing our builders to push builds from our different docker images,
> but as it's just a string, I'm hoping that the name build-environment
> will work for other use cases.
> There is already a compiler field which is populated by the backend
> from the build.json and some simple parsing.
>
> There is a matching PR [2] to kernelci-core-staging to add a
> build-environment string to our builds (example:
> docker-kernelci/build-gcc-7_arm64) but it has been suggested this
> could be shortened to 'gcc-7' which I'm happy to change.
>

It sounds like the 2 PRs work hand-in-hand, so they'll need to be
tested and merged together.

Potentially, we could have different flavours of the same
compiler version from different build environments.  Say, maybe
Linaro's GCC 8 will be different from Debian's etc...  Will the
build environment be shown in the email reports, or will there
still just be the compiler version as we do now?


> For the frontend, this change "just works", because build.py is now
> populating the file_server_resource field, all the URLs on the build
> information pages will be correct.
>

Great, let's take this opportunity to try and avoid having to
repeat the magic URL format everywhere.

Same question as with the email reports though, will the build
environment information be visible anywhere or will we solely
rely on the compiler version?

I would really like to get this deployed to staging for testing and
> then rolled out ASAP as it's a requirement for starting builds with
> compilers other than GCC, and multiple versions of GCC.
>

Sounds like something to look at next week now that all fixes
seem to be available in PRs.

I guess one extra change would be to iterate over several build
environments for each architecture, but that would probably rely
on proper build configurations (i.e. using a build-configs.yaml
like we do with test-configs.yaml).

Guillaume

[1] https://github.com/kernelci/kernelci-backend/pull/86
> [2] https://github.com/kernelci/kernelci-core-staging/pull/43
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 4101 bytes --]

Re: [kernelci] Multiple compiler support in the backend

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 915 bytes --]

On Fri, Dec 14, 2018 at 03:01:04PM +0000, Guillaume Tucker wrote:

> Potentially, we could have different flavours of the same
> compiler version from different build environments.  Say, maybe
> Linaro's GCC 8 will be different from Debian's etc...  Will the
> build environment be shown in the email reports, or will there
> still just be the compiler version as we do now?

At least for GCC based toolchains the convention is for vendors to
modify the version number of the compiler to indicate that they've added
stuff - for example, on my desktop right now GCC reports that it is "gcc
(Debian 8.2.0-6) 8.2.0", the Linaro toolchains do the same as do other
binary toolchains I've looked at in the past.  That's probably enough I
think?

Sadly the clang-7 packages for Debian don't seem to have anything like
that but my locally built copies of clang-8 have the git hash and repo
URL for everything in the build.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [kernelci] Multiple compiler support in the backend

[Kevin Hilman]

"Mark Brown" <broonie@kernel.org> writes:

> On Fri, Dec 14, 2018 at 03:01:04PM +0000, Guillaume Tucker wrote:
>
>> Potentially, we could have different flavours of the same
>> compiler version from different build environments.  Say, maybe
>> Linaro's GCC 8 will be different from Debian's etc...  Will the
>> build environment be shown in the email reports, or will there
>> still just be the compiler version as we do now?
>
> At least for GCC based toolchains the convention is for vendors to
> modify the version number of the compiler to indicate that they've added
> stuff - for example, on my desktop right now GCC reports that it is "gcc
> (Debian 8.2.0-6) 8.2.0", the Linaro toolchains do the same as do other
> binary toolchains I've looked at in the past.  That's probably enough I
> think?
>
> Sadly the clang-7 packages for Debian don't seem to have anything like
> that but my locally built copies of clang-8 have the git hash and repo
> URL for everything in the build.

The debian packages for recent clang builds I'm using are from
apt.llvm.org, and they seem to provide pretty detailed versioning:

   $ clang --version |head -1
   clang version 8.0.0-svn345496-1~exp1+0~20181029105533.852~1.gbpf10f36 (trunk)

Kevin

Re: [kernelci] Multiple compiler support in the backend

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

On Fri, Dec 14, 2018 at 10:48:24AM -0800, Kevin Hilman wrote:
> "Mark Brown" <broonie@kernel.org> writes:

> > Sadly the clang-7 packages for Debian don't seem to have anything like
> > that but my locally built copies of clang-8 have the git hash and repo
> > URL for everything in the build.

> The debian packages for recent clang builds I'm using are from
> apt.llvm.org, and they seem to provide pretty detailed versioning:

>    $ clang --version |head -1
>    clang version 8.0.0-svn345496-1~exp1+0~20181029105533.852~1.gbpf10f36 (trunk)

Yeah, that looks like a svn version of what I got from my local git
snapshot - the Debian packages will be built from tarballs so they lack
the VCS information, that's going to be a problem with releases I guess.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]