FreeBSD build regressions

FreeBSD build regressions

[Alex Bennée]

Hi,

It looks like the build has been broken on Cirrus since at least 7b2c4c:

  https://cirrus-ci.com/github/qemu/qemu

I did attempt to have a look but "vm-build-freebsd" seems to be failing
with a different error:

  10:31:47  [alex.bennee@hackbox2:~/l/q/b/all] master|✔ + make vm-build-freebsd
    GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 meson dtc capstone slirp
      VM-BUILD freebsd
  cross containers  no

  NOTE: guest cross-compilers enabled: cc cc
  ld-elf.so.1: /usr/local/lib/libpython3.7m.so.1.0: Undefined symbol "close_range@FBSD_1.6"
  ld-elf.so.1: /usr/local/lib/libpython3.7m.so.1.0: Undefined symbol "close_range@FBSD_1.6"
  The Meson build system
  Version: 0.55.3
  Source dir: /usr/home/qemu/qemu-test.egp8wG/src
  Build dir: /usr/home/qemu/qemu-test.egp8wG/build
  Build type: native build
  Project name: qemu
  Project version: 5.2.50
  ld-elf.so.1: /usr/local/lib/libpython3.7m.so.1.0: Undefined symbol "close_range@FBSD_1.6"

  ../src/meson.build:1:0: ERROR: Executables created by c compiler cc are not runnable.

  A full log can be found at /usr/home/qemu/qemu-test.egp8wG/build/meson-logs/meson-log.txt

  ERROR: meson setup failed

  /home/alex.bennee/lsrc/qemu.git/tests/vm/Makefile.include:95: recipe for target 'vm-build-freebsd' failed
  make: *** [vm-build-freebsd] Error 3

Tracking back to before the previously mentioned commit it was still
failing which makes me think something has happened to the BSD image (or
something is missing since the build changes). I'd appreciate it if
someone with more FreeBSD knowledge can look into both regressions
because frankly I find it exhausting enough tracking down Linux
regressions when they occur.

Thanks,

-- 
Alex Bennée

Re: FreeBSD build regressions

[Peter Maydell]

On Fri, 19 Feb 2021 at 10:39, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Hi,
>
> It looks like the build has been broken on Cirrus since at least 7b2c4c:
>
>   https://cirrus-ci.com/github/qemu/qemu
>
> I did attempt to have a look but "vm-build-freebsd" seems to be failing
> with a different error

FWIW the vm-build-freebsd build-and-test works for me, as I
continue to run it as part of the merge tests. Is this something
to do with whether you already have a freebsd image cached
as opposed to it getting re-built from scratch (perhaps with
a newer FreeBSD)?

-- PMM

Re: FreeBSD build regressions

[Daniel P. Berrangé]

On Fri, Feb 19, 2021 at 10:29:50AM +0000, Alex Bennée wrote:
> 
> Hi,
> 
> It looks like the build has been broken on Cirrus since at least 7b2c4c:
> 
>   https://cirrus-ci.com/github/qemu/qemu
> 
> I did attempt to have a look but "vm-build-freebsd" seems to be failing
> with a different error:
> 
>   10:31:47  [alex.bennee@hackbox2:~/l/q/b/all] master|✔ + make vm-build-freebsd
>     GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 meson dtc capstone slirp
>       VM-BUILD freebsd
>   cross containers  no
> 
>   NOTE: guest cross-compilers enabled: cc cc
>   ld-elf.so.1: /usr/local/lib/libpython3.7m.so.1.0: Undefined symbol "close_range@FBSD_1.6"
>   ld-elf.so.1: /usr/local/lib/libpython3.7m.so.1.0: Undefined symbol "close_range@FBSD_1.6"
>   The Meson build system
>   Version: 0.55.3
>   Source dir: /usr/home/qemu/qemu-test.egp8wG/src
>   Build dir: /usr/home/qemu/qemu-test.egp8wG/build
>   Build type: native build
>   Project name: qemu
>   Project version: 5.2.50
>   ld-elf.so.1: /usr/local/lib/libpython3.7m.so.1.0: Undefined symbol "close_range@FBSD_1.6"
> 
>   ../src/meson.build:1:0: ERROR: Executables created by c compiler cc are not runnable.
> 
>   A full log can be found at /usr/home/qemu/qemu-test.egp8wG/build/meson-logs/meson-log.txt
> 
>   ERROR: meson setup failed
> 
>   /home/alex.bennee/lsrc/qemu.git/tests/vm/Makefile.include:95: recipe for target 'vm-build-freebsd' failed
>   make: *** [vm-build-freebsd] Error 3
> 
> Tracking back to before the previously mentioned commit it was still
> failing which makes me think something has happened to the BSD image (or
> something is missing since the build changes). I'd appreciate it if
> someone with more FreeBSD knowledge can look into both regressions
> because frankly I find it exhausting enough tracking down Linux
> regressions when they occur.

Yes, this is a problem with the image in Cirrus, which hit libvirt
CI too. We "fixed" it by changing to the freebsd-12-2  instead of
freebsd-12-1 image.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: FreeBSD build regressions

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Fri, 19 Feb 2021 at 10:39, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Hi,
>>
>> It looks like the build has been broken on Cirrus since at least 7b2c4c:
>>
>>   https://cirrus-ci.com/github/qemu/qemu
>>
>> I did attempt to have a look but "vm-build-freebsd" seems to be failing
>> with a different error
>
> FWIW the vm-build-freebsd build-and-test works for me, as I
> continue to run it as part of the merge tests. Is this something
> to do with whether you already have a freebsd image cached
> as opposed to it getting re-built from scratch (perhaps with
> a newer FreeBSD)?

It did re-run the installation when I first called the target so I guess
it was that.

-- 
Alex Bennée

Re: FreeBSD build regressions

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

On Fri, Feb 19, 2021, 3:59 AM Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
> > On Fri, 19 Feb 2021 at 10:39, Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >>
> >>
> >> Hi,
> >>
> >> It looks like the build has been broken on Cirrus since at least 7b2c4c:
> >>
> >>   https://cirrus-ci.com/github/qemu/qemu
> >>
> >> I did attempt to have a look but "vm-build-freebsd" seems to be failing
> >> with a different error
> >
> > FWIW the vm-build-freebsd build-and-test works for me, as I
> > continue to run it as part of the merge tests. Is this something
> > to do with whether you already have a freebsd image cached
> > as opposed to it getting re-built from scratch (perhaps with
> > a newer FreeBSD)?
>
> It did re-run the installation when I first called the target so I guess
> it was that.
>

Python was built against a newer FreeBSD that had symbols that weren't in
12.1. Where did the python build come from?

Warner

-- 
> Alex Bennée
>
>

[-- Attachment #2: Type: text/html, Size: 2015 bytes --]

Re: FreeBSD build regressions

[Gerd Hoffmann]

On Fri, Feb 19, 2021 at 10:41:44AM +0000, Peter Maydell wrote:
> On Fri, 19 Feb 2021 at 10:39, Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> >
> > Hi,
> >
> > It looks like the build has been broken on Cirrus since at least 7b2c4c:
> >
> >   https://cirrus-ci.com/github/qemu/qemu
> >
> > I did attempt to have a look but "vm-build-freebsd" seems to be failing
> > with a different error
> 
> FWIW the vm-build-freebsd build-and-test works for me, as I
> continue to run it as part of the merge tests. Is this something
> to do with whether you already have a freebsd image cached
> as opposed to it getting re-built from scratch (perhaps with
> a newer FreeBSD)?

The base image should be the same no matter what (updating that needs a
tests/vm/freebsd update which in turn triggers a rebuild).  The addon
package versions may differ though, so in case a broken package enters
the freebsd package repos it may happen that old, existing vm images
continue to work whereas newly created images don't ...

Trying to rebuild the freebsd image here results in this:

[ ... ]
### Installing packages ...
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.16.1...
Newer FreeBSD version for package pkg:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1202000                          <- freebsd 12.2 expected ?
- running kernel: 1201000                   <- freebsd 12.1 running ?
Ignore the mismatch and continue? [y/N]: 

So it seems the freebsd 12.1 images tries to fetch 12.2 packages when
running "pkg install -y <list>", which would explain why they don't
work.

Switching to freebsd 12.2 should solve this, at least until 12.3 is
released, but I'm wondering why the freebsd pkg utility fetches
incompatible packages in the first place and whenever there is any
way to avoid this ...

take care,
  Gerd

Re: FreeBSD build regressions

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 3660 bytes --]

First off, sorry for the hassles this caused... The problem is a failure to
upgrade for reasons explained below... While the reasons make sense, it's
still a bit of a hassle, for which I apologize.

On Fri, Feb 19, 2021 at 8:51 AM Gerd Hoffmann <gerd@kraxel.org> wrote:

> On Fri, Feb 19, 2021 at 10:41:44AM +0000, Peter Maydell wrote:
> > On Fri, 19 Feb 2021 at 10:39, Alex Bennée <alex.bennee@linaro.org>
> wrote:
> > >
> > >
> > > Hi,
> > >
> > > It looks like the build has been broken on Cirrus since at least
> 7b2c4c:
> > >
> > >   https://cirrus-ci.com/github/qemu/qemu
> > >
> > > I did attempt to have a look but "vm-build-freebsd" seems to be failing
> > > with a different error
> >
> > FWIW the vm-build-freebsd build-and-test works for me, as I
> > continue to run it as part of the merge tests. Is this something
> > to do with whether you already have a freebsd image cached
> > as opposed to it getting re-built from scratch (perhaps with
> > a newer FreeBSD)?
>
> The base image should be the same no matter what (updating that needs a
> tests/vm/freebsd update which in turn triggers a rebuild).  The addon
> package versions may differ though, so in case a broken package enters
> the freebsd package repos it may happen that old, existing vm images
> continue to work whereas newly created images don't ...
>
> Trying to rebuild the freebsd image here results in this:
>
> [ ... ]
> ### Installing packages ...
> Bootstrapping pkg from pkg+
> http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
> Verifying signature with trusted certificate pkg.freebsd.org.2013102301...
> done
> Installing pkg-1.16.1...
> Newer FreeBSD version for package pkg:
> To ignore this error set IGNORE_OSVERSION=yes
> - package: 1202000                          <- freebsd 12.2 expected ?
> - running kernel: 1201000                   <- freebsd 12.1 running ?
> Ignore the mismatch and continue? [y/N]:
>
> So it seems the freebsd 12.1 images tries to fetch 12.2 packages when
> running "pkg install -y <list>", which would explain why they don't
> work.
>

FreeBSD 12.1 images fetch the FreeBSD 12 packages. This works until FreeBSD
12.1 hits end of life and the packages are built on newer versions. There's
no long-term support for releases once a new minor release happens. The
economics of this quickly get out of hand as the testing matrix explodes
and the number of machines needed to support all old versions (even on
supported branches) would be prohibitively expensive.


> Switching to freebsd 12.2 should solve this, at least until 12.3 is
> released, but I'm wondering why the freebsd pkg utility fetches
> incompatible packages in the first place and whenever there is any
> way to avoid this ...
>

FreeBSD builds packages on the oldest supported version in the stable
branch. Due to forward compatibility, that means all supported versions of
FreeBSD 12.x will work. Recently, FreeBSD 12.1 became unsupported, so the
build machines clicked forward to 12.2. Since there's no 'forward
compatibility' guarantees, this problem was hit. While you can run binaries
compiled on old versions of the software on new versions of the system, you
can't necessarily do the inverse because new symbols are introduced (in
this case close_range).

The base FreeBSD image should be rolled when new versions are released to
avoid this issue. tests/vm/freebsd should be updated. I have a full
morning, but if nobody has beat me to it, I'll submit a patch this
afternoon to fix this and add it to my list of things to do when FreeBSD
creates a new release.

Warner

[-- Attachment #2: Type: text/html, Size: 4658 bytes --]

Re: FreeBSD build regressions

[Peter Maydell]

On Fri, 19 Feb 2021 at 16:08, Warner Losh <imp@bsdimp.com> wrote:
> FreeBSD builds packages on the oldest supported version in the stable branch. Due to forward compatibility, that means all supported versions of FreeBSD 12.x will work. Recently, FreeBSD 12.1 became unsupported, so the build machines clicked forward to 12.2. Since there's no 'forward compatibility' guarantees, this problem was hit. While you can run binaries compiled on old versions of the software on new versions of the system, you can't necessarily do the inverse because new symbols are introduced (in this case close_range).

It makes perfect sense that you don't want to support older
versions forever and that at some point newer packages aren't
valid on old systems, but I don't understand why an
older 12.1 system then says "but I'm going to go ahead and
install these won't-work packages anyway" rather than
"oh dear, I'm out of support, there are no newer packages
available, I will install whatever the last archived version
of the package for my OS version is" (or "I will install nothing").
I'm surprised this doesn't break a lot of real-world users...

-- PMM

Re: FreeBSD build regressions

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2039 bytes --]

On Fri, Feb 19, 2021 at 9:14 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Fri, 19 Feb 2021 at 16:08, Warner Losh <imp@bsdimp.com> wrote:
> > FreeBSD builds packages on the oldest supported version in the stable
> branch. Due to forward compatibility, that means all supported versions of
> FreeBSD 12.x will work. Recently, FreeBSD 12.1 became unsupported, so the
> build machines clicked forward to 12.2. Since there's no 'forward
> compatibility' guarantees, this problem was hit. While you can run binaries
> compiled on old versions of the software on new versions of the system, you
> can't necessarily do the inverse because new symbols are introduced (in
> this case close_range).
>
> It makes perfect sense that you don't want to support older
> versions forever and that at some point newer packages aren't
> valid on old systems, but I don't understand why an
> older 12.1 system then says "but I'm going to go ahead and
> install these won't-work packages anyway" rather than
> "oh dear, I'm out of support, there are no newer packages
> available, I will install whatever the last archived version
> of the package for my OS version is" (or "I will install nothing").
> I'm surprised this doesn't break a lot of real-world users...
>

That's a reasonable expectation. I'd kinda expected that to be the default,
but it looks like it might not be. I'll see if I can get the freebsd vm
updated to use something safer and/or work with the pkg folks to get it to
do the safe thing here if there's no easy way to do this with command line
/ config settings. I think the issue is that we set IGNORE_OSVERSION which
is needed for the case when we were running 12.0 packages on 12.1, but it's
harmful for this case. This highlights, I think, a rough edge in pkg.

Short term, I'll bump things up to 12.2 which will take care of the
immediate issue. I should have a patch by later in the day.... I may also
have a patch to detect the mismatch directly and report it until this issue
can be resolved in FreeBSD's pkg.

Warner

[-- Attachment #2: Type: text/html, Size: 2608 bytes --]