MSYS2 and libfdt

MSYS2 and libfdt

[Thomas Huth]

  Hi all,

in some spare minutes, I started playing with a patch to try to remove the 
dtc submodule from the QEMU git repository - according to 
https://repology.org/project/dtc/versions our supported build platforms 
should now all provide the minimum required version.

However, I'm hitting a problem with Windows / MSYS2 in the CI jobs: The 
libfdt is packaged as part of the dtc package there:

  https://packages.msys2.org/package/dtc

... meaning that it is added with a usr/include and usr/lib path prefix 
instead of mingw64/include and mingw64/lib like other packages are using 
(see e.g. 
https://packages.msys2.org/package/mingw-w64-x86_64-zlib?repo=mingw64). Thus 
the compiler does not find the library there. Also there does not seem to be 
a difference between a i686 (32-bit) and x86_64 (64-bit) variant available 
here? Does anybody know how libfdt is supposed to be used with MSYS2 ?

  Thomas

Re: MSYS2 and libfdt

[Stefan Weil via]

Am 19.01.23 um 09:14 schrieb Thomas Huth:

>
>  Hi all,
>
> in some spare minutes, I started playing with a patch to try to remove 
> the dtc submodule from the QEMU git repository - according to 
> https://repology.org/project/dtc/versions our supported build 
> platforms should now all provide the minimum required version.
>
> However, I'm hitting a problem with Windows / MSYS2 in the CI jobs: 
> The libfdt is packaged as part of the dtc package there:
>
>  https://packages.msys2.org/package/dtc
>
> ... meaning that it is added with a usr/include and usr/lib path 
> prefix instead of mingw64/include and mingw64/lib like other packages 
> are using (see e.g. 
> https://packages.msys2.org/package/mingw-w64-x86_64-zlib?repo=mingw64). 
> Thus the compiler does not find the library there. Also there does not 
> seem to be a difference between a i686 (32-bit) and x86_64 (64-bit) 
> variant available here? Does anybody know how libfdt is supposed to be 
> used with MSYS2 ?
>
>  Thomas


Hi Thomas,

"dtc" is not the right package for cross builds. We'd require 
mingw-w64-i686-dtc and mingw-w64-x86_64-dtc packages for the QEMU build, 
but those packages are currently not provided by MSYS2.

See https://packages.msys2.org/search?t=binpkg&q=zlib for the zlib 
packages and related cross build packages.

Stefan

Re: MSYS2 and libfdt

[Marc-André Lureau]

Hi

On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>
>
>   Hi all,
>
> in some spare minutes, I started playing with a patch to try to remove the
> dtc submodule from the QEMU git repository - according to
> https://repology.org/project/dtc/versions our supported build platforms
> should now all provide the minimum required version.
>
> However, I'm hitting a problem with Windows / MSYS2 in the CI jobs: The
> libfdt is packaged as part of the dtc package there:
>
>   https://packages.msys2.org/package/dtc
>
> ... meaning that it is added with a usr/include and usr/lib path prefix
> instead of mingw64/include and mingw64/lib like other packages are using
> (see e.g.
> https://packages.msys2.org/package/mingw-w64-x86_64-zlib?repo=mingw64). Thus
> the compiler does not find the library there. Also there does not seem to be
> a difference between a i686 (32-bit) and x86_64 (64-bit) variant available
> here? Does anybody know how libfdt is supposed to be used with MSYS2 ?

The msys environment is a bit special, it's not an environment for a
particular build target, my understanding is that it holds common
files/tools.

dtc should be added to https://github.com/msys2/MINGW-packages for it
to be available as a target dependency.

-- 
Marc-André Lureau

Re: MSYS2 and libfdt

[Thomas Huth]

On 19/01/2023 09.55, Stefan Weil wrote:
> Am 19.01.23 um 09:14 schrieb Thomas Huth:
> 
>>
>>  Hi all,
>>
>> in some spare minutes, I started playing with a patch to try to remove the 
>> dtc submodule from the QEMU git repository - according to 
>> https://repology.org/project/dtc/versions our supported build platforms 
>> should now all provide the minimum required version.
>>
>> However, I'm hitting a problem with Windows / MSYS2 in the CI jobs: The 
>> libfdt is packaged as part of the dtc package there:
>>
>>  https://packages.msys2.org/package/dtc
>>
>> ... meaning that it is added with a usr/include and usr/lib path prefix 
>> instead of mingw64/include and mingw64/lib like other packages are using 
>> (see e.g. 
>> https://packages.msys2.org/package/mingw-w64-x86_64-zlib?repo=mingw64). 
>> Thus the compiler does not find the library there. Also there does not 
>> seem to be a difference between a i686 (32-bit) and x86_64 (64-bit) 
>> variant available here? Does anybody know how libfdt is supposed to be 
>> used with MSYS2 ?
>>
>>  Thomas
> 
> 
> Hi Thomas,
> 
> "dtc" is not the right package for cross builds. We'd require 
> mingw-w64-i686-dtc and mingw-w64-x86_64-dtc packages for the QEMU build, but 
> those packages are currently not provided by MSYS2.

Ok, thanks ... so that basically means we cannot get rid of the dtc 
submodule yet.

  Thomas

Re: MSYS2 and libfdt

[Thomas Huth]

On 19/01/2023 09.56, Marc-André Lureau wrote:
> Hi
> 
> On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>>
>>
>>    Hi all,
>>
>> in some spare minutes, I started playing with a patch to try to remove the
>> dtc submodule from the QEMU git repository - according to
>> https://repology.org/project/dtc/versions our supported build platforms
>> should now all provide the minimum required version.
>>
>> However, I'm hitting a problem with Windows / MSYS2 in the CI jobs: The
>> libfdt is packaged as part of the dtc package there:
>>
>>    https://packages.msys2.org/package/dtc
>>
>> ... meaning that it is added with a usr/include and usr/lib path prefix
>> instead of mingw64/include and mingw64/lib like other packages are using
>> (see e.g.
>> https://packages.msys2.org/package/mingw-w64-x86_64-zlib?repo=mingw64). Thus
>> the compiler does not find the library there. Also there does not seem to be
>> a difference between a i686 (32-bit) and x86_64 (64-bit) variant available
>> here? Does anybody know how libfdt is supposed to be used with MSYS2 ?
> 
> The msys environment is a bit special, it's not an environment for a
> particular build target, my understanding is that it holds common
> files/tools.
> 
> dtc should be added to https://github.com/msys2/MINGW-packages for it
> to be available as a target dependency.

Do you have already any experience in requesting a new package there? Could 
you maybe do it? ... since I don't have a proper MinGW installation here, it 
would be very cumbersome for me right now.

  Thomas

Re: MSYS2 and libfdt

[Marc-André Lureau]

Hi Thomas

On Fri, Jan 20, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 19/01/2023 09.56, Marc-André Lureau wrote:
> > Hi
> >
> > On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
> >>
> >>
> >>    Hi all,
> >>
> >> in some spare minutes, I started playing with a patch to try to remove the
> >> dtc submodule from the QEMU git repository - according to
> >> https://repology.org/project/dtc/versions our supported build platforms
> >> should now all provide the minimum required version.
> >>
> >> However, I'm hitting a problem with Windows / MSYS2 in the CI jobs: The
> >> libfdt is packaged as part of the dtc package there:
> >>
> >>    https://packages.msys2.org/package/dtc
> >>
> >> ... meaning that it is added with a usr/include and usr/lib path prefix
> >> instead of mingw64/include and mingw64/lib like other packages are using
> >> (see e.g.
> >> https://packages.msys2.org/package/mingw-w64-x86_64-zlib?repo=mingw64). Thus
> >> the compiler does not find the library there. Also there does not seem to be
> >> a difference between a i686 (32-bit) and x86_64 (64-bit) variant available
> >> here? Does anybody know how libfdt is supposed to be used with MSYS2 ?
> >
> > The msys environment is a bit special, it's not an environment for a
> > particular build target, my understanding is that it holds common
> > files/tools.
> >
> > dtc should be added to https://github.com/msys2/MINGW-packages for it
> > to be available as a target dependency.
>
> Do you have already any experience in requesting a new package there? Could
> you maybe do it? ... since I don't have a proper MinGW installation here, it
> would be very cumbersome for me right now.
>

Here you go (although let see what CI has to say):
https://github.com/msys2/MINGW-packages/pull/15168

The msys2 maintainers are usually very quick and helpful, in my experience.

(I sometime use a windev evaluation VM, that I import with the help of
https://github.com/elmarco/virt-install-windev)

-- 
Marc-André Lureau

Re: MSYS2 and libfdt

[Daniel P. Berrangé]

On Fri, Jan 20, 2023 at 05:57:29PM +0400, Marc-André Lureau wrote:
> Hi Thomas
> 
> On Fri, Jan 20, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 19/01/2023 09.56, Marc-André Lureau wrote:
> > > Hi
> > >
> > > On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
> > >>
> > >>
> > >>    Hi all,
> > >>
> > >> in some spare minutes, I started playing with a patch to try to remove the
> > >> dtc submodule from the QEMU git repository - according to
> > >> https://repology.org/project/dtc/versions our supported build platforms
> > >> should now all provide the minimum required version.
> > >>
> > >> However, I'm hitting a problem with Windows / MSYS2 in the CI jobs: The
> > >> libfdt is packaged as part of the dtc package there:
> > >>
> > >>    https://packages.msys2.org/package/dtc
> > >>
> > >> ... meaning that it is added with a usr/include and usr/lib path prefix
> > >> instead of mingw64/include and mingw64/lib like other packages are using
> > >> (see e.g.
> > >> https://packages.msys2.org/package/mingw-w64-x86_64-zlib?repo=mingw64). Thus
> > >> the compiler does not find the library there. Also there does not seem to be
> > >> a difference between a i686 (32-bit) and x86_64 (64-bit) variant available
> > >> here? Does anybody know how libfdt is supposed to be used with MSYS2 ?
> > >
> > > The msys environment is a bit special, it's not an environment for a
> > > particular build target, my understanding is that it holds common
> > > files/tools.
> > >
> > > dtc should be added to https://github.com/msys2/MINGW-packages for it
> > > to be available as a target dependency.
> >
> > Do you have already any experience in requesting a new package there? Could
> > you maybe do it? ... since I don't have a proper MinGW installation here, it
> > would be very cumbersome for me right now.
> >
> 
> Here you go (although let see what CI has to say):
> https://github.com/msys2/MINGW-packages/pull/15168
> 
> The msys2 maintainers are usually very quick and helpful, in my experience.

And it merged 1 day after you posted it. So yes, the msys2 maintainers
are indeed very quick & helpful :-)

So in theory we can try to drop the submodule for dtc now


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: MSYS2 and libfdt

[Thomas Huth]

On 23/01/2023 17.23, Daniel P. Berrangé wrote:
> On Fri, Jan 20, 2023 at 05:57:29PM +0400, Marc-André Lureau wrote:
>> Hi Thomas
>>
>> On Fri, Jan 20, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>>>
>>> On 19/01/2023 09.56, Marc-André Lureau wrote:
>>>> Hi
>>>>
>>>> On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>>>>>
>>>>>     Hi all,
>>>>>
>>>>> in some spare minutes, I started playing with a patch to try to remove the
>>>>> dtc submodule from the QEMU git repository - according to
>>>>> https://repology.org/project/dtc/versions our supported build platforms
>>>>> should now all provide the minimum required version.
>>>>>
>>>>> However, I'm hitting a problem with Windows / MSYS2 in the CI jobs: The
>>>>> libfdt is packaged as part of the dtc package there:
>>>>>
>>>>>     https://packages.msys2.org/package/dtc
>>>>>
>>>>> ... meaning that it is added with a usr/include and usr/lib path prefix
>>>>> instead of mingw64/include and mingw64/lib like other packages are using
>>>>> (see e.g.
>>>>> https://packages.msys2.org/package/mingw-w64-x86_64-zlib?repo=mingw64). Thus
>>>>> the compiler does not find the library there. Also there does not seem to be
>>>>> a difference between a i686 (32-bit) and x86_64 (64-bit) variant available
>>>>> here? Does anybody know how libfdt is supposed to be used with MSYS2 ?
>>>>
>>>> The msys environment is a bit special, it's not an environment for a
>>>> particular build target, my understanding is that it holds common
>>>> files/tools.
>>>>
>>>> dtc should be added to https://github.com/msys2/MINGW-packages for it
>>>> to be available as a target dependency.
>>>
>>> Do you have already any experience in requesting a new package there? Could
>>> you maybe do it? ... since I don't have a proper MinGW installation here, it
>>> would be very cumbersome for me right now.
>>>
>>
>> Here you go (although let see what CI has to say):
>> https://github.com/msys2/MINGW-packages/pull/15168
>>
>> The msys2 maintainers are usually very quick and helpful, in my experience.

Great, I just gave it a try, and it seems to be working, indeed:

https://gitlab.com/thuth/qemu/-/jobs/3649509495#L668

Thank you very much Marc-André and Biswapriyo!

> And it merged 1 day after you posted it. So yes, the msys2 maintainers
> are indeed very quick & helpful :-)
> 
> So in theory we can try to drop the submodule for dtc now

Ok, I'll give my patch another try to see whether all the other systems have 
a usable version of libfdt available, too.

  Thomas

NetBSD and libfdt (was: Re: MSYS2 and libfdt)

[Thomas Huth]

On 24/01/2023 10.20, Thomas Huth wrote:
[...]
>>>>> On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>>>>>>
>>>>>>     Hi all,
>>>>>>
>>>>>> in some spare minutes, I started playing with a patch to try to remove 
>>>>>> the
>>>>>> dtc submodule from the QEMU git repository - according to
>>>>>> https://repology.org/project/dtc/versions our supported build platforms
>>>>>> should now all provide the minimum required version.
[...]
> Ok, I'll give my patch another try to see whether all the other systems have 
> a usable version of libfdt available, too.

... and I apparently missed NetBSD in my first research: Looks like NetBSD 
is still using dtc v1.4.7 which is too old for QEMU. (though 
https://www.netbsd.org/docs/software/3rdparty/ talks about v1.5.1, I only 
get dtc 1.4.7 in our NetBSD VM).

Reinoud, Ryo, any chance that you could get dtc updated to a newer version 
(at least 1.5.1) in NetBSD ?

  Thanks,
   Thomas

MinGW and libfdt (was: Re: MSYS2 and libfdt)

[Thomas Huth]

On 23/01/2023 17.23, Daniel P. Berrangé wrote:
> On Fri, Jan 20, 2023 at 05:57:29PM +0400, Marc-André Lureau wrote:
...
>>>> On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>>>>>
>>>>>
>>>>>     Hi all,
>>>>>
>>>>> in some spare minutes, I started playing with a patch to try to remove the
>>>>> dtc submodule from the QEMU git repository - according to
>>>>> https://repology.org/project/dtc/versions our supported build platforms
>>>>> should now all provide the minimum required version.
...
> So in theory we can try to drop the submodule for dtc now

The dtc package is also still missing in the MinGW cross compiler suite in 
Fedora ... does anybody know what's the right way to request it there?

  Thomas

Re: MinGW and libfdt (was: Re: MSYS2 and libfdt)

[Daniel P. Berrangé]

On Tue, Jan 24, 2023 at 03:43:25PM +0100, Thomas Huth wrote:
> On 23/01/2023 17.23, Daniel P. Berrangé wrote:
> > On Fri, Jan 20, 2023 at 05:57:29PM +0400, Marc-André Lureau wrote:
> ...
> > > > > On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
> > > > > > 
> > > > > > 
> > > > > >     Hi all,
> > > > > > 
> > > > > > in some spare minutes, I started playing with a patch to try to remove the
> > > > > > dtc submodule from the QEMU git repository - according to
> > > > > > https://repology.org/project/dtc/versions our supported build platforms
> > > > > > should now all provide the minimum required version.
> ...
> > So in theory we can try to drop the submodule for dtc now
> 
> The dtc package is also still missing in the MinGW cross compiler suite in
> Fedora ... does anybody know what's the right way to request it there?

Someone will need to write a specfile, and submit it for review. I can do
the submission, or the review, but not both (can't mark your own homework)


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: MinGW and libfdt (was: Re: MSYS2 and libfdt)

[Marc-André Lureau]

Hi

On Tue, Jan 24, 2023 at 7:08 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jan 24, 2023 at 03:43:25PM +0100, Thomas Huth wrote:
> > On 23/01/2023 17.23, Daniel P. Berrangé wrote:
> > > On Fri, Jan 20, 2023 at 05:57:29PM +0400, Marc-André Lureau wrote:
> > ...
> > > > > > On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
> > > > > > >
> > > > > > >
> > > > > > >     Hi all,
> > > > > > >
> > > > > > > in some spare minutes, I started playing with a patch to try to remove the
> > > > > > > dtc submodule from the QEMU git repository - according to
> > > > > > > https://repology.org/project/dtc/versions our supported build platforms
> > > > > > > should now all provide the minimum required version.
> > ...
> > > So in theory we can try to drop the submodule for dtc now
> >
> > The dtc package is also still missing in the MinGW cross compiler suite in
> > Fedora ... does anybody know what's the right way to request it there?
>
> Someone will need to write a specfile, and submit it for review. I can do
> the submission, or the review, but not both (can't mark your own homework)
>

It's already been in rawhide for a few months. We can probably merge
and update f37.
https://packages.fedoraproject.org/pkgs/dtc/dtc/


-- 
Marc-André Lureau

Re: MinGW and libfdt (was: Re: MSYS2 and libfdt)

[Thomas Huth]

On 24/01/2023 20.10, Marc-André Lureau wrote:
> Hi
> 
> On Tue, Jan 24, 2023 at 7:08 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>>
>> On Tue, Jan 24, 2023 at 03:43:25PM +0100, Thomas Huth wrote:
>>> On 23/01/2023 17.23, Daniel P. Berrangé wrote:
>>>> On Fri, Jan 20, 2023 at 05:57:29PM +0400, Marc-André Lureau wrote:
>>> ...
>>>>>>> On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>      Hi all,
>>>>>>>>
>>>>>>>> in some spare minutes, I started playing with a patch to try to remove the
>>>>>>>> dtc submodule from the QEMU git repository - according to
>>>>>>>> https://repology.org/project/dtc/versions our supported build platforms
>>>>>>>> should now all provide the minimum required version.
>>> ...
>>>> So in theory we can try to drop the submodule for dtc now
>>>
>>> The dtc package is also still missing in the MinGW cross compiler suite in
>>> Fedora ... does anybody know what's the right way to request it there?
>>
>> Someone will need to write a specfile, and submit it for review. I can do
>> the submission, or the review, but not both (can't mark your own homework)
>>
> 
> It's already been in rawhide for a few months. We can probably merge
> and update f37.
> https://packages.fedoraproject.org/pkgs/dtc/dtc/

Ah, great, good to know that it is on its way already. Even if this only 
gets released in F38 this still should be fine ... F38 is not that far away 
anymore.

  Thomas

Re: NetBSD and libfdt (was: Re: MSYS2 and libfdt)

[Nick Hudson]

hi,

On 24/01/2023 11:27, Thomas Huth wrote:
> On 24/01/2023 10.20, Thomas Huth wrote:
> [...]
>>>>>> On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> 
>>>>>> wrote:
>>>>>>>
>>>>>>>     Hi all,
>>>>>>>
>>>>>>> in some spare minutes, I started playing with a patch to try to 
>>>>>>> remove the
>>>>>>> dtc submodule from the QEMU git repository - according to
>>>>>>> https://repology.org/project/dtc/versions our supported build 
>>>>>>> platforms
>>>>>>> should now all provide the minimum required version.
> [...]
>> Ok, I'll give my patch another try to see whether all the other 
>> systems have a usable version of libfdt available, too.
> 
> ... and I apparently missed NetBSD in my first research: Looks like 
> NetBSD is still using dtc v1.4.7 which is too old for QEMU. (though 
> https://www.netbsd.org/docs/software/3rdparty/ talks about v1.5.1, I 
> only get dtc 1.4.7 in our NetBSD VM).

The not yet released netbsd-10 and -current have 1.5.1. Perhaps you can 
use netbsd-10 for your VM?

Thanks,
Nick

Re: NetBSD and libfdt

[Thomas Huth]

On 25/01/2023 12.36, Nick Hudson wrote:
> hi,
> 
> On 24/01/2023 11:27, Thomas Huth wrote:
>> On 24/01/2023 10.20, Thomas Huth wrote:
>> [...]
>>>>>>> On Thu, Jan 19, 2023 at 12:31 PM Thomas Huth <thuth@redhat.com> wrote:
>>>>>>>>
>>>>>>>>     Hi all,
>>>>>>>>
>>>>>>>> in some spare minutes, I started playing with a patch to try to 
>>>>>>>> remove the
>>>>>>>> dtc submodule from the QEMU git repository - according to
>>>>>>>> https://repology.org/project/dtc/versions our supported build platforms
>>>>>>>> should now all provide the minimum required version.
>> [...]
>>> Ok, I'll give my patch another try to see whether all the other systems 
>>> have a usable version of libfdt available, too.
>>
>> ... and I apparently missed NetBSD in my first research: Looks like NetBSD 
>> is still using dtc v1.4.7 which is too old for QEMU. (though 
>> https://www.netbsd.org/docs/software/3rdparty/ talks about v1.5.1, I only 
>> get dtc 1.4.7 in our NetBSD VM).
> 
> The not yet released netbsd-10 and -current have 1.5.1. Perhaps you can use 
> netbsd-10 for your VM?

Ah, ok, thanks, I didn't know yet that NetBSD 10 is just about to get 
released (hopefully?) soon...
Anyway, even if it gets released this year, it means we have to carry the 
dtc submodule in QEMU for two more years according to our support policy:

 
https://www.qemu.org/docs/master/about/build-platforms.html#linux-os-macos-freebsd-netbsd-openbsd

(since NetBSD will be supported by QEMU for two more years)

Will there be another minor release of NetBSD 9 where it would be possible 
to update dtc to at least version 1.5.1, too?

  Thanks,
   Thomas