[Qemu-devel] Yet another git submodule rant

[Qemu-devel] Yet another git submodule rant

[Thomas Huth]

That automatic git submodule stuff now broke my workflow again. I
usually keep the git repository on my laptop and then simply rsync the
sources (without .git directories) to my target machine to compile it
there. Used to work great for years. Now it's broken, the build process
complains:

GIT submodule checkout is out of date. Please run
  scripts/git-submodule.sh update
from the source directory checkout /home/thuth/devel/qemu

Running "scripts/git-submodule.sh update" did not fix the issue at all -
I first had to tinker with it for a while to find out that I simply have
to delete ".git-submodule-status" in my git tree to fix the issue.

I've got the feeling that all this submodule crap is constantly causing
pain ... do we really need this? Can't we find another solution instead?
Or at least stop modifying files automatically in the $SRC_PATH ?

... ok, sorry for the harsh words, ... but now that I've written this
down, I feel at least a little bit better ...

 Thomas

Re: [Qemu-devel] Yet another git submodule rant

[Daniel P. Berrange]

On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:
> 
> That automatic git submodule stuff now broke my workflow again. I
> usually keep the git repository on my laptop and then simply rsync the
> sources (without .git directories) to my target machine to compile it
> there. Used to work great for years. Now it's broken, the build process
> complains:

Excluding the .git dir will be a problem - that needs to exist, unless
you are building from tar.xz (where we will bundled all the submodule
sources together in the tar.xz).

If you really want to exclude the .git directories, then instead of
doing an rsync, then use the scripts/archive-source.sh tool to create
an archive of all the sources (which bundles required submodules)
and unpack that on your target machine. Then all submodule handling
on the target will be skipped.

> 
> GIT submodule checkout is out of date. Please run
>   scripts/git-submodule.sh update
> from the source directory checkout /home/thuth/devel/qemu
> 
> Running "scripts/git-submodule.sh update" did not fix the issue at all -
> I first had to tinker with it for a while to find out that I simply have
> to delete ".git-submodule-status" in my git tree to fix the issue.

I think it got confused from you deleting the .git directory

> I've got the feeling that all this submodule crap is constantly causing
> pain ... do we really need this? Can't we find another solution instead?
> Or at least stop modifying files automatically in the $SRC_PATH ?

We *have* to modify SRC_PATH, as that's where git submodule checkouts
live. If we didn't do that, then people would end up having failed
build due to submodules not existing, or even worse, having a silently
incorrect build due to using the wrong checked out version.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Yet another git submodule rant

[Gerd Hoffmann]

On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:
> 
> That automatic git submodule stuff now broke my workflow again. I
> usually keep the git repository on my laptop and then simply rsync the
> sources (without .git directories) to my target machine to compile it
> there.

How about excluding all files listed in .gitignore from rsync?  Which I
guess would be a good idea anyway because you don't want sync over the
build artifacts to the target machine if you want compile there.  Might
be as simple as "rsync --exclude-from .gitignore" (didn't test though).

Or syncronize using git instead of rsync.  This is what I am doing.

cheers,
  Gerd

Re: [Qemu-devel] Yet another git submodule rant

[Thomas Huth]

On 08.11.2017 11:53, Gerd Hoffmann wrote:
> On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:
>>
>> That automatic git submodule stuff now broke my workflow again. I
>> usually keep the git repository on my laptop and then simply rsync the
>> sources (without .git directories) to my target machine to compile it
>> there.
> 
> How about excluding all files listed in .gitignore from rsync?  Which I
> guess would be a good idea anyway because you don't want sync over the
> build artifacts to the target machine if you want compile there.  Might
> be as simple as "rsync --exclude-from .gitignore" (didn't test though).

That's a neat idea, thanks, I'll give it a try!

> Or syncronize using git instead of rsync.  This is what I am doing.

Tried that in the past already, but it didn't really feel right (I don't
want to check in each time before syncing my files)...

 Thomas

Re: [Qemu-devel] Yet another git submodule rant

[Michal Suchánek]

On Wed, 8 Nov 2017 10:25:29 +0000
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:
> > 
> > That automatic git submodule stuff now broke my workflow again. I
> > usually keep the git repository on my laptop and then simply rsync
> > the sources (without .git directories) to my target machine to
> > compile it there. Used to work great for years. Now it's broken,
> > the build process complains:  
> 
> Excluding the .git dir will be a problem - that needs to exist, unless
> you are building from tar.xz (where we will bundled all the submodule
> sources together in the tar.xz).
> 
> If you really want to exclude the .git directories, then instead of
> doing an rsync, then use the scripts/archive-source.sh tool to create
> an archive of all the sources (which bundles required submodules)
> and unpack that on your target machine. Then all submodule handling
> on the target will be skipped.

This probably does not work because it archives committed sources and
not actual sources.

> 
> > 
> > GIT submodule checkout is out of date. Please run
> >   scripts/git-submodule.sh update
> > from the source directory checkout /home/thuth/devel/qemu
> > 
> > Running "scripts/git-submodule.sh update" did not fix the issue at
> > all - I first had to tinker with it for a while to find out that I
> > simply have to delete ".git-submodule-status" in my git tree to fix
> > the issue.  
> 
> I think it got confused from you deleting the .git directory
> 
> > I've got the feeling that all this submodule crap is constantly
> > causing pain ... do we really need this? Can't we find another
> > solution instead? Or at least stop modifying files automatically in
> > the $SRC_PATH ?  
> 
> We *have* to modify SRC_PATH, as that's where git submodule checkouts
> live. If we didn't do that, then people would end up having failed
> build due to submodules not existing, or even worse, having a silently
> incorrect build due to using the wrong checked out version.

And that's another reason why submodules are bad thing.

So again, can this be done without submodules?

Let somebody else do the testing and polishing of this pre-alpha git
feature.

Thanks

Michal

Re: [Qemu-devel] Yet another git submodule rant

[Philippe Mathieu-Daudé]

On 11/08/2017 06:57 AM, Thomas Huth wrote:
> 
> That automatic git submodule stuff now broke my workflow again. I
> usually keep the git repository on my laptop and then simply rsync the
> sources (without .git directories) to my target machine to compile it
> there. Used to work great for years. Now it's broken, the build process
> complains:
> 
> GIT submodule checkout is out of date. Please run
>   scripts/git-submodule.sh update
> from the source directory checkout /home/thuth/devel/qemu
> 
> Running "scripts/git-submodule.sh update" did not fix the issue at all -
> I first had to tinker with it for a while to find out that I simply have
> to delete ".git-submodule-status" in my git tree to fix the issue.
> 
> I've got the feeling that all this submodule crap is constantly causing
> pain ... do we really need this? Can't we find another solution instead?
> Or at least stop modifying files automatically in the $SRC_PATH ?

Also yesterday on IRC:

<RaV3N> [...] I downloaded the qemu source from git and tried to compile
it. I am getting this:

./configure --static && make && sudo make install
 CC      ui/input-keymap.o
ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
No such file or directory

<cota> [...] you might want to do "git submodule init && git submodule
update"

Re: [Qemu-devel] Yet another git submodule rant

[Daniel P. Berrange]

On Wed, Nov 08, 2017 at 01:05:27PM +0100, Michal Suchánek wrote:
> On Wed, 8 Nov 2017 10:25:29 +0000
> "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:
> > > 
> > > That automatic git submodule stuff now broke my workflow again. I
> > > usually keep the git repository on my laptop and then simply rsync
> > > the sources (without .git directories) to my target machine to
> > > compile it there. Used to work great for years. Now it's broken,
> > > the build process complains:  
> > 
> > Excluding the .git dir will be a problem - that needs to exist, unless
> > you are building from tar.xz (where we will bundled all the submodule
> > sources together in the tar.xz).
> > 
> > If you really want to exclude the .git directories, then instead of
> > doing an rsync, then use the scripts/archive-source.sh tool to create
> > an archive of all the sources (which bundles required submodules)
> > and unpack that on your target machine. Then all submodule handling
> > on the target will be skipped.
> 
> This probably does not work because it archives committed sources and
> not actual sources.

Actually that's not quite correct - if it sees you have non-comitted
changes, and will use 'git stash create' to capture them in the archive
it creates.  So it would only miss brand new files which have not yet
been 'git add'ed

> > > I've got the feeling that all this submodule crap is constantly
> > > causing pain ... do we really need this? Can't we find another
> > > solution instead? Or at least stop modifying files automatically in
> > > the $SRC_PATH ?  
> > 
> > We *have* to modify SRC_PATH, as that's where git submodule checkouts
> > live. If we didn't do that, then people would end up having failed
> > build due to submodules not existing, or even worse, having a silently
> > incorrect build due to using the wrong checked out version.
> 
> And that's another reason why submodules are bad thing.
> 
> So again, can this be done without submodules?
> 
> Let somebody else do the testing and polishing of this pre-alpha git
> feature.

Honestly I think the submodule stuff is working pretty well - a handful
of people have mentioned problems and we've addressed most of them or
illustrated alternative approaches to deal with it. For most developers
it has been pretty much transparent and just works, or we would have
seen alot more noise.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Yet another git submodule rant

[Michal Suchánek]

On Wed, 8 Nov 2017 12:34:17 +0000
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> On Wed, Nov 08, 2017 at 01:05:27PM +0100, Michal Suchánek wrote:
> > On Wed, 8 Nov 2017 10:25:29 +0000
> > "Daniel P. Berrange" <berrange@redhat.com> wrote:
> >   
> > > On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:  
> > > > 
> > > > That automatic git submodule stuff now broke my workflow again.
> > > > I usually keep the git repository on my laptop and then simply
> > > > rsync the sources (without .git directories) to my target
> > > > machine to compile it there. Used to work great for years. Now
> > > > it's broken, the build process complains:    
> > > 
> > > Excluding the .git dir will be a problem - that needs to exist,
> > > unless you are building from tar.xz (where we will bundled all
> > > the submodule sources together in the tar.xz).
> > > 
> > > If you really want to exclude the .git directories, then instead
> > > of doing an rsync, then use the scripts/archive-source.sh tool to
> > > create an archive of all the sources (which bundles required
> > > submodules) and unpack that on your target machine. Then all
> > > submodule handling on the target will be skipped.  
> > 
> > This probably does not work because it archives committed sources
> > and not actual sources.  
> 
> Actually that's not quite correct - if it sees you have non-comitted
> changes, and will use 'git stash create' to capture them in the
> archive it creates.  So it would only miss brand new files which have
> not yet been 'git add'ed

So there really isn't a good solution.

> 
> > > > I've got the feeling that all this submodule crap is constantly
> > > > causing pain ... do we really need this? Can't we find another
> > > > solution instead? Or at least stop modifying files
> > > > automatically in the $SRC_PATH ?    
> > > 
> > > We *have* to modify SRC_PATH, as that's where git submodule
> > > checkouts live. If we didn't do that, then people would end up
> > > having failed build due to submodules not existing, or even
> > > worse, having a silently incorrect build due to using the wrong
> > > checked out version.  
> > 
> > And that's another reason why submodules are bad thing.
> > 
> > So again, can this be done without submodules?
> > 
> > Let somebody else do the testing and polishing of this pre-alpha git
> > feature.  
> 
> Honestly I think the submodule stuff is working pretty well - a
> handful of people have mentioned problems and we've addressed most of
> them or illustrated alternative approaches to deal with it. For most
> developers it has been pretty much transparent and just works, or we
> would have seen alot more noise.

> On 11/08/2017 06:57 AM, Thomas Huth wrote:

> 
> Also yesterday on IRC:
> 
> <RaV3N> [...] I downloaded the qemu source from git and tried to
> compile it. I am getting this:
> 
> ./configure --static && make && sudo make install
>  CC      ui/input-keymap.o
> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
> No such file or directory
> 
> <cota> [...] you might want to do "git submodule init && git submodule
> update"
> 

 .. you call that transparent.

Yeah, right.

Please don't pretend that something is working when it clearly isn't.

Repositories using submodules are far from first class citizens as far
as support from git is concerned.

Therefore projects that don't have git development at least as
secondary focus should not use them until that changes.

Thanks

Michal

Re: [Qemu-devel] Yet another git submodule rant

[Daniel P. Berrange]

On Wed, Nov 08, 2017 at 01:43:15PM +0100, Michal Suchánek wrote:
> On Wed, 8 Nov 2017 12:34:17 +0000
> "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > On Wed, Nov 08, 2017 at 01:05:27PM +0100, Michal Suchánek wrote:
> > > On Wed, 8 Nov 2017 10:25:29 +0000
> > > "Daniel P. Berrange" <berrange@redhat.com> wrote:
> > >   
> > > > On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:  
> > > > > 
> > > > > That automatic git submodule stuff now broke my workflow again.
> > > > > I usually keep the git repository on my laptop and then simply
> > > > > rsync the sources (without .git directories) to my target
> > > > > machine to compile it there. Used to work great for years. Now
> > > > > it's broken, the build process complains:    
> > > > 
> > > > Excluding the .git dir will be a problem - that needs to exist,
> > > > unless you are building from tar.xz (where we will bundled all
> > > > the submodule sources together in the tar.xz).
> > > > 
> > > > If you really want to exclude the .git directories, then instead
> > > > of doing an rsync, then use the scripts/archive-source.sh tool to
> > > > create an archive of all the sources (which bundles required
> > > > submodules) and unpack that on your target machine. Then all
> > > > submodule handling on the target will be skipped.  
> > > 
> > > This probably does not work because it archives committed sources
> > > and not actual sources.  
> > 
> > Actually that's not quite correct - if it sees you have non-comitted
> > changes, and will use 'git stash create' to capture them in the
> > archive it creates.  So it would only miss brand new files which have
> > not yet been 'git add'ed
> 
> So there really isn't a good solution.
> 
> > 
> > > > > I've got the feeling that all this submodule crap is constantly
> > > > > causing pain ... do we really need this? Can't we find another
> > > > > solution instead? Or at least stop modifying files
> > > > > automatically in the $SRC_PATH ?    
> > > > 
> > > > We *have* to modify SRC_PATH, as that's where git submodule
> > > > checkouts live. If we didn't do that, then people would end up
> > > > having failed build due to submodules not existing, or even
> > > > worse, having a silently incorrect build due to using the wrong
> > > > checked out version.  
> > > 
> > > And that's another reason why submodules are bad thing.
> > > 
> > > So again, can this be done without submodules?
> > > 
> > > Let somebody else do the testing and polishing of this pre-alpha git
> > > feature.  
> > 
> > Honestly I think the submodule stuff is working pretty well - a
> > handful of people have mentioned problems and we've addressed most of
> > them or illustrated alternative approaches to deal with it. For most
> > developers it has been pretty much transparent and just works, or we
> > would have seen alot more noise.
> 
> > On 11/08/2017 06:57 AM, Thomas Huth wrote:
> 
> > 
> > Also yesterday on IRC:
> > 
> > <RaV3N> [...] I downloaded the qemu source from git and tried to
> > compile it. I am getting this:
> > 
> > ./configure --static && make && sudo make install
> >  CC      ui/input-keymap.o
> > ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
> > No such file or directory
> > 
> > <cota> [...] you might want to do "git submodule init && git submodule
> > update"
> > 
> 
>  .. you call that transparent.
> 
> Yeah, right.
> 
> Please don't pretend that something is working when it clearly isn't.
> 
> Repositories using submodules are far from first class citizens as far
> as support from git is concerned.
> 
> Therefore projects that don't have git development at least as
> secondary focus should not use them until that changes.

That's really overstating the problem. There are some caveats with use of
submodules, but they are hardly showstoppers that prevent their usage
entirely. There are plenty of projects using GIT with submodules for
*years* (libvirt has used them for 8 years).

Some of the problems pointed out have been clear bugs in the way we've
integrated into QEMU's build system, which I've been addressing. Others
have been due to usage scenarios I didn't anticipate, which I've attempted
to provide solutions for, either by improving the build system, or suggesting
alternative approaches. We'll continue to try to improve this, as we would
for any other major change to the buildsystem we've been through, which
caused problems for some workflows. I none the less believe this is working
for the vast majority of developers of QEMU, given the size of our community
vs the number of problems reported.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Yet another git submodule rant

[Daniel P. Berrange]

On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
> On 11/08/2017 06:57 AM, Thomas Huth wrote:
> > 
> > That automatic git submodule stuff now broke my workflow again. I
> > usually keep the git repository on my laptop and then simply rsync the
> > sources (without .git directories) to my target machine to compile it
> > there. Used to work great for years. Now it's broken, the build process
> > complains:
> > 
> > GIT submodule checkout is out of date. Please run
> >   scripts/git-submodule.sh update
> > from the source directory checkout /home/thuth/devel/qemu
> > 
> > Running "scripts/git-submodule.sh update" did not fix the issue at all -
> > I first had to tinker with it for a while to find out that I simply have
> > to delete ".git-submodule-status" in my git tree to fix the issue.
> > 
> > I've got the feeling that all this submodule crap is constantly causing
> > pain ... do we really need this? Can't we find another solution instead?
> > Or at least stop modifying files automatically in the $SRC_PATH ?
> 
> Also yesterday on IRC:
> 
> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
> it. I am getting this:
> 
> ./configure --static && make && sudo make install
>  CC      ui/input-keymap.o
> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
> No such file or directory

I had a pull request merged yesterday later afternoon which possibly
would address that problem, though hard hard to say for certain.

> <cota> [...] you might want to do "git submodule init && git submodule
> update"

This certainly should not be required - the build system should always
take care of that, so it would be considered a bug if it didn't happen.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Yet another git submodule rant

[Philippe Mathieu-Daudé]

On 11/08/2017 08:48 AM, Thomas Huth wrote:
> On 08.11.2017 11:53, Gerd Hoffmann wrote:
>> On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:
>>>
>>> That automatic git submodule stuff now broke my workflow again. I
>>> usually keep the git repository on my laptop and then simply rsync the
>>> sources (without .git directories) to my target machine to compile it
>>> there.
>>
>> How about excluding all files listed in .gitignore from rsync?  Which I
>> guess would be a good idea anyway because you don't want sync over the
>> build artifacts to the target machine if you want compile there.  Might
>> be as simple as "rsync --exclude-from .gitignore" (didn't test though).
> 
> That's a neat idea, thanks, I'll give it a try!
> 
>> Or syncronize using git instead of rsync.  This is what I am doing.
> 
> Tried that in the past already, but it didn't really feel right (I don't
> want to check in each time before syncing my files)...

TIL: git push --recurse-submodules=on-demand

GIT-PUSH(1)

       --recurse-submodules=check|on-demand|only|no

           [...] If check is used Git will verify that all submodule
           commits that changed in the revisions to be pushed are
           available on at least one remote of the submodule. If any
           commits are missing the push will be aborted and exit with
           non-zero status. If on-demand is used all submodules that
           changed in the revisions to be pushed will be pushed. If
           on-demand was not able to push all necessary revisions it
           will also be aborted and exit with non-zero status. [...]

GIT-CONFIG(1)

       push.recurseSubmodules
           Make sure all submodule commits used by the revisions to be
           pushed are available on a remote-tracking branch. If the
           value is check then Git will verify that all submodule
           commits that changed in the revisions to be pushed are
           available on at least one remote of the submodule. If any
           commits are missing, the push will be aborted and exit with
           non-zero status. If the value is on-demand then all
           submodules that changed in the revisions to be pushed will
           be pushed. If on-demand was not able to push all necessary
           revisions it will also be aborted and exit with non-zero
           status. [...]

Re: [Qemu-devel] Yet another git submodule rant

[Laszlo Ersek]

On 11/08/17 11:53, Gerd Hoffmann wrote:
> On Wed, Nov 08, 2017 at 10:57:21AM +0100, Thomas Huth wrote:
>>
>> That automatic git submodule stuff now broke my workflow again. I
>> usually keep the git repository on my laptop and then simply rsync the
>> sources (without .git directories) to my target machine to compile it
>> there.
> 
> How about excluding all files listed in .gitignore from rsync?  Which I
> guess would be a good idea anyway because you don't want sync over the
> build artifacts to the target machine if you want compile there.  Might
> be as simple as "rsync --exclude-from .gitignore" (didn't test though).
> 
> Or syncronize using git instead of rsync.  This is what I am doing.

Me too,

$ git push --mirror host:dir

(I don't mind having to commit first.)

Thanks,
Laszlo

Re: [Qemu-devel] Yet another git submodule rant

[Alexey Kardashevskiy]

On 08/11/17 20:57, Thomas Huth wrote:
> 
> That automatic git submodule stuff now broke my workflow again. I
> usually keep the git repository on my laptop and then simply rsync the
> sources (without .git directories) to my target machine to compile it
> there. Used to work great for years. Now it's broken, the build process
> complains:
> 
> GIT submodule checkout is out of date. Please run
>   scripts/git-submodule.sh update
> from the source directory checkout /home/thuth/devel/qemu
> 
> Running "scripts/git-submodule.sh update" did not fix the issue at all -
> I first had to tinker with it for a while to find out that I simply have
> to delete ".git-submodule-status" in my git tree to fix the issue.
> 
> I've got the feeling that all this submodule crap is constantly causing
> pain ... do we really need this? Can't we find another solution instead?
> Or at least stop modifying files automatically in the $SRC_PATH ?
> 
> ... ok, sorry for the harsh words, ... but now that I've written this
> down, I feel at least a little bit better ...


Since there is resistance to add a way to disable changing $SRC_PATH, I
guess the way to go is simply move .git folder outside of source tree and
use $GIT_DIR or --work-dir only when you want git to manage the sources -
this should make $git_submodules empty in ./configure and git-submodule.sh
not to touch anything.


-- 
Alexey

Re: [Qemu-devel] Yet another git submodule rant

[Alexey Kardashevskiy]

On 09/11/17 00:01, Daniel P. Berrange wrote:
> On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
>> On 11/08/2017 06:57 AM, Thomas Huth wrote:
>>>
>>> That automatic git submodule stuff now broke my workflow again. I
>>> usually keep the git repository on my laptop and then simply rsync the
>>> sources (without .git directories) to my target machine to compile it
>>> there. Used to work great for years. Now it's broken, the build process
>>> complains:
>>>
>>> GIT submodule checkout is out of date. Please run
>>>   scripts/git-submodule.sh update
>>> from the source directory checkout /home/thuth/devel/qemu
>>>
>>> Running "scripts/git-submodule.sh update" did not fix the issue at all -
>>> I first had to tinker with it for a while to find out that I simply have
>>> to delete ".git-submodule-status" in my git tree to fix the issue.
>>>
>>> I've got the feeling that all this submodule crap is constantly causing
>>> pain ... do we really need this? Can't we find another solution instead?
>>> Or at least stop modifying files automatically in the $SRC_PATH ?
>>
>> Also yesterday on IRC:
>>
>> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
>> it. I am getting this:
>>
>> ./configure --static && make && sudo make install
>>  CC      ui/input-keymap.o
>> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
>> No such file or directory
> 
> I had a pull request merged yesterday later afternoon which possibly
> would address that problem, though hard hard to say for certain.

wow, already? :(

I still wonder why do not we checkout submodules into the build directory
and why .git-submodule-status is not there too...



-- 
Alexey

Re: [Qemu-devel] Yet another git submodule rant

[Daniel P. Berrange]

On Fri, Nov 10, 2017 at 09:35:54PM +1100, Alexey Kardashevskiy wrote:
> On 09/11/17 00:01, Daniel P. Berrange wrote:
> > On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
> >> On 11/08/2017 06:57 AM, Thomas Huth wrote:
> >>>
> >>> That automatic git submodule stuff now broke my workflow again. I
> >>> usually keep the git repository on my laptop and then simply rsync the
> >>> sources (without .git directories) to my target machine to compile it
> >>> there. Used to work great for years. Now it's broken, the build process
> >>> complains:
> >>>
> >>> GIT submodule checkout is out of date. Please run
> >>>   scripts/git-submodule.sh update
> >>> from the source directory checkout /home/thuth/devel/qemu
> >>>
> >>> Running "scripts/git-submodule.sh update" did not fix the issue at all -
> >>> I first had to tinker with it for a while to find out that I simply have
> >>> to delete ".git-submodule-status" in my git tree to fix the issue.
> >>>
> >>> I've got the feeling that all this submodule crap is constantly causing
> >>> pain ... do we really need this? Can't we find another solution instead?
> >>> Or at least stop modifying files automatically in the $SRC_PATH ?
> >>
> >> Also yesterday on IRC:
> >>
> >> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
> >> it. I am getting this:
> >>
> >> ./configure --static && make && sudo make install
> >>  CC      ui/input-keymap.o
> >> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
> >> No such file or directory
> > 
> > I had a pull request merged yesterday later afternoon which possibly
> > would address that problem, though hard hard to say for certain.
> 
> wow, already? :(
> 
> I still wonder why do not we checkout submodules into the build directory
> and why .git-submodule-status is not there too...

That simply isn't the way submodules work, they are inherently part of
the source tree, and the status file reflects that too.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Yet another git submodule rant

[Alexey Kardashevskiy]

On 10/11/17 21:41, Daniel P. Berrange wrote:
> On Fri, Nov 10, 2017 at 09:35:54PM +1100, Alexey Kardashevskiy wrote:
>> On 09/11/17 00:01, Daniel P. Berrange wrote:
>>> On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
>>>> On 11/08/2017 06:57 AM, Thomas Huth wrote:
>>>>>
>>>>> That automatic git submodule stuff now broke my workflow again. I
>>>>> usually keep the git repository on my laptop and then simply rsync the
>>>>> sources (without .git directories) to my target machine to compile it
>>>>> there. Used to work great for years. Now it's broken, the build process
>>>>> complains:
>>>>>
>>>>> GIT submodule checkout is out of date. Please run
>>>>>   scripts/git-submodule.sh update
>>>>> from the source directory checkout /home/thuth/devel/qemu
>>>>>
>>>>> Running "scripts/git-submodule.sh update" did not fix the issue at all -
>>>>> I first had to tinker with it for a while to find out that I simply have
>>>>> to delete ".git-submodule-status" in my git tree to fix the issue.
>>>>>
>>>>> I've got the feeling that all this submodule crap is constantly causing
>>>>> pain ... do we really need this? Can't we find another solution instead?
>>>>> Or at least stop modifying files automatically in the $SRC_PATH ?
>>>>
>>>> Also yesterday on IRC:
>>>>
>>>> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
>>>> it. I am getting this:
>>>>
>>>> ./configure --static && make && sudo make install
>>>>  CC      ui/input-keymap.o
>>>> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
>>>> No such file or directory
>>>
>>> I had a pull request merged yesterday later afternoon which possibly
>>> would address that problem, though hard hard to say for certain.
>>
>> wow, already? :(
>>
>> I still wonder why do not we checkout submodules into the build directory
>> and why .git-submodule-status is not there too...
> 
> That simply isn't the way submodules work, they are inherently part of
> the source tree, and the status file reflects that too.

Sorry, I am missing the point here. What precisely does prevent us from
checking out the required modules to the build directory and build them
there? git provides a submodule repository url and sha1 for the current
qemu branch.

Yes, the submodule tree inside the qemu tree might be different from the
one in the build directory but the purpose of all of this is to always
build the correct code - so this requirement is met. And it will still be
better than changing the $SRC_PATH when a user specifically asked not to do
that by calling "./configure --source-path='.



-- 
Alexey

Re: [Qemu-devel] Yet another git submodule rant

[Peter Maydell]

On 10 November 2017 at 13:46, Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> And it will still be
> better than changing the $SRC_PATH when a user specifically asked not to do
> that by calling "./configure --source-path='.

I'm not terribly happy with the submodule stuff either, but
that configure rune is not making use of a documented
or supported feature. --source-path is for telling configure
where the source code is. Passing it an empty string should
probably be rejected as an error.

thanks
-- PMM

Re: [Qemu-devel] Yet another git submodule rant

[Daniel P. Berrange]

On Sat, Nov 11, 2017 at 12:46:36AM +1100, Alexey Kardashevskiy wrote:
> On 10/11/17 21:41, Daniel P. Berrange wrote:
> > On Fri, Nov 10, 2017 at 09:35:54PM +1100, Alexey Kardashevskiy wrote:
> >> On 09/11/17 00:01, Daniel P. Berrange wrote:
> >>> On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
> >>>> On 11/08/2017 06:57 AM, Thomas Huth wrote:
> >>>>>
> >>>>> That automatic git submodule stuff now broke my workflow again. I
> >>>>> usually keep the git repository on my laptop and then simply rsync the
> >>>>> sources (without .git directories) to my target machine to compile it
> >>>>> there. Used to work great for years. Now it's broken, the build process
> >>>>> complains:
> >>>>>
> >>>>> GIT submodule checkout is out of date. Please run
> >>>>>   scripts/git-submodule.sh update
> >>>>> from the source directory checkout /home/thuth/devel/qemu
> >>>>>
> >>>>> Running "scripts/git-submodule.sh update" did not fix the issue at all -
> >>>>> I first had to tinker with it for a while to find out that I simply have
> >>>>> to delete ".git-submodule-status" in my git tree to fix the issue.
> >>>>>
> >>>>> I've got the feeling that all this submodule crap is constantly causing
> >>>>> pain ... do we really need this? Can't we find another solution instead?
> >>>>> Or at least stop modifying files automatically in the $SRC_PATH ?
> >>>>
> >>>> Also yesterday on IRC:
> >>>>
> >>>> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
> >>>> it. I am getting this:
> >>>>
> >>>> ./configure --static && make && sudo make install
> >>>>  CC      ui/input-keymap.o
> >>>> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
> >>>> No such file or directory
> >>>
> >>> I had a pull request merged yesterday later afternoon which possibly
> >>> would address that problem, though hard hard to say for certain.
> >>
> >> wow, already? :(
> >>
> >> I still wonder why do not we checkout submodules into the build directory
> >> and why .git-submodule-status is not there too...
> > 
> > That simply isn't the way submodules work, they are inherently part of
> > the source tree, and the status file reflects that too.
> 
> Sorry, I am missing the point here. What precisely does prevent us from
> checking out the required modules to the build directory and build them
> there? git provides a submodule repository url and sha1 for the current
> qemu branch.

The build directory should never contain any of your version controlled
source, as that will get irretrievably lost when the build dir is purged.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Yet another git submodule rant

[Alexey Kardashevskiy]

On 11/11/17 01:01, Peter Maydell wrote:
> On 10 November 2017 at 13:46, Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
>> And it will still be
>> better than changing the $SRC_PATH when a user specifically asked not to do
>> that by calling "./configure --source-path='.
> 
> I'm not terribly happy with the submodule stuff either, but
> that configure rune is not making use of a documented
> or supported feature. --source-path is for telling configure
> where the source code is. Passing it an empty string should
> probably be rejected as an error.


It is not empty, I just cut an example too much :) The script looks like:

cd /home/aik/pbuild/qemu-garrison2-ppc64/
/home/aik/p/qemu/configure --target-list=ppc64-softmmu
--source-path=/home/aik/p/qemu/



-- 
Alexey

Re: [Qemu-devel] Yet another git submodule rant

[Alexey Kardashevskiy]

On 11/11/17 01:22, Daniel P. Berrange wrote:
> On Sat, Nov 11, 2017 at 12:46:36AM +1100, Alexey Kardashevskiy wrote:
>> On 10/11/17 21:41, Daniel P. Berrange wrote:
>>> On Fri, Nov 10, 2017 at 09:35:54PM +1100, Alexey Kardashevskiy wrote:
>>>> On 09/11/17 00:01, Daniel P. Berrange wrote:
>>>>> On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
>>>>>> On 11/08/2017 06:57 AM, Thomas Huth wrote:
>>>>>>>
>>>>>>> That automatic git submodule stuff now broke my workflow again. I
>>>>>>> usually keep the git repository on my laptop and then simply rsync the
>>>>>>> sources (without .git directories) to my target machine to compile it
>>>>>>> there. Used to work great for years. Now it's broken, the build process
>>>>>>> complains:
>>>>>>>
>>>>>>> GIT submodule checkout is out of date. Please run
>>>>>>>   scripts/git-submodule.sh update
>>>>>>> from the source directory checkout /home/thuth/devel/qemu
>>>>>>>
>>>>>>> Running "scripts/git-submodule.sh update" did not fix the issue at all -
>>>>>>> I first had to tinker with it for a while to find out that I simply have
>>>>>>> to delete ".git-submodule-status" in my git tree to fix the issue.
>>>>>>>
>>>>>>> I've got the feeling that all this submodule crap is constantly causing
>>>>>>> pain ... do we really need this? Can't we find another solution instead?
>>>>>>> Or at least stop modifying files automatically in the $SRC_PATH ?
>>>>>>
>>>>>> Also yesterday on IRC:
>>>>>>
>>>>>> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
>>>>>> it. I am getting this:
>>>>>>
>>>>>> ./configure --static && make && sudo make install
>>>>>>  CC      ui/input-keymap.o
>>>>>> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
>>>>>> No such file or directory
>>>>>
>>>>> I had a pull request merged yesterday later afternoon which possibly
>>>>> would address that problem, though hard hard to say for certain.
>>>>
>>>> wow, already? :(
>>>>
>>>> I still wonder why do not we checkout submodules into the build directory
>>>> and why .git-submodule-status is not there too...
>>>
>>> That simply isn't the way submodules work, they are inherently part of
>>> the source tree, and the status file reflects that too.
>>
>> Sorry, I am missing the point here. What precisely does prevent us from
>> checking out the required modules to the build directory and build them
>> there? git provides a submodule repository url and sha1 for the current
>> qemu branch.
> 
> The build directory should never contain any of your version controlled
> source, as that will get irretrievably lost when the build dir is purged.

I am not suggesting editing submodule sources from the build directory, I
am suggesting not building them from the source directory, in order to keep
them in sync, Makefile can rsync or "git push" them to the build directory.
Yeah, fairly ugly but still more correct than the current solution.

And again, .git-submodule-status is not a source file, what is it doing in
$SRC_PATH?

Now I cannot compile qemu pretty much 50% of time because of that
enhancement. In the today's episode:

ssh aikhostos2 cd /home/aik/pbuild/qemu-aikhostos2-ppc64/ ;
/home/aik/p/qemu/configure --target-list=ppc64-softmmu
--source-path=/home/aik/p/qemu/ --enable-debug --enable-debug-info
--disable-werror --disable-git-update --enable-trace-backend=log

All good. One detail:
capstone          git

ssh aikhostos2 make -C /home/aik/pbuild/qemu-aikhostos2-ppc64/ -j24

make: Entering directory `/home/aik/pbuild/qemu-aikhostos2-ppc64'
  GEN     ppc64-softmmu/config-devices.mak.tmp
mkdir -p dtc/libfdt
mkdir -p dtc/tests
  GEN     config-host.h
  GEN     qemu-options.def
make[1]: *** No rule to make target
`/home/aik/pbuild/qemu-aikhostos2-ppc64/capstone/libcapstone.a'.  Stop.
make: *** [subdir-capstone] Error 2
make: *** Waiting for unfinished jobs....
  GEN     ppc64-softmmu/config-devices.mak


How, why, where are all these nice warning messages? Ah, there they are
(added 'set -x' to the script):

+ substat=.git-submodule-status
+ command=status
+ shift
+ maybe_modules='ui/keycodemapdb dtc capstone'
+ test -z git
+ modules=
+ for m in '$maybe_modules'
+ git submodule status ui/keycodemapdb
+ test 128 = 0
+ echo 'warn: ignoring non-existent submodule ui/keycodemapdb'
+ for m in '$maybe_modules'
+ git submodule status dtc
+ test 128 = 0:m
+ echo 'warn: ignoring non-existent submodule dtc'
+ for m in '$maybe_modules'
+ git submodule status capstone
+ test 128 = 0
+ echo 'warn: ignoring non-existent submodule capstone'
+ test -n 'ui/keycodemapdb dtc capstone'
+ test -e .git
+ case "$command" in
+ test -z 'ui/keycodemapdb dtc capstone'
+ test -f .git-submodule-status
+ exit 1


I assume that 'warn' is not printed because of Makefile's 'quiet-command'.

And this fails because on the server the source directory is created with
'git worktree' which points to bare git repo outside of the source tree and
the git on that build machine has no idea about worktrees (the repo is
visible to the build machine).

[vpl1 ~]$ ssh -x aikhostos2 'cd p/qemu && pwd && git --version && ls -d
../qemu.git && git submodule status'
/home/aik/p/qemu
git version 1.8.3.1
../qemu.git
fatal: Not a git repository: /home/aik/p/qemu.git/worktrees/qemu


Again - ok, when I know what is the problem, I know how to fix it - simply
move .git away when running ./configure and make (as it also runs
./configure some time) - but the messages... Firstly ./configure decides
that 'capstone' comes from 'git', then it simply fails with no indication
why it went wrong.

Ha. I just found another workaround - I can do ./configure --with-git=false
and silently disable git updates. Pretty much pointing a gun to my leg,
ready to shoot.

We could just check git submodule status every build and print a warning if
it is out of sync but continue. This way I would look at the submodules
list and decide what to do.


-- 
Alexey

Re: [Qemu-devel] Yet another git submodule rant

[Alexey Kardashevskiy]

On 11/11/17 12:10, Alexey Kardashevskiy wrote:
> On 11/11/17 01:22, Daniel P. Berrange wrote:
>> On Sat, Nov 11, 2017 at 12:46:36AM +1100, Alexey Kardashevskiy wrote:
>>> On 10/11/17 21:41, Daniel P. Berrange wrote:
>>>> On Fri, Nov 10, 2017 at 09:35:54PM +1100, Alexey Kardashevskiy wrote:
>>>>> On 09/11/17 00:01, Daniel P. Berrange wrote:
>>>>>> On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
>>>>>>> On 11/08/2017 06:57 AM, Thomas Huth wrote:
>>>>>>>>
>>>>>>>> That automatic git submodule stuff now broke my workflow again. I
>>>>>>>> usually keep the git repository on my laptop and then simply rsync the
>>>>>>>> sources (without .git directories) to my target machine to compile it
>>>>>>>> there. Used to work great for years. Now it's broken, the build process
>>>>>>>> complains:
>>>>>>>>
>>>>>>>> GIT submodule checkout is out of date. Please run
>>>>>>>>   scripts/git-submodule.sh update
>>>>>>>> from the source directory checkout /home/thuth/devel/qemu
>>>>>>>>
>>>>>>>> Running "scripts/git-submodule.sh update" did not fix the issue at all -
>>>>>>>> I first had to tinker with it for a while to find out that I simply have
>>>>>>>> to delete ".git-submodule-status" in my git tree to fix the issue.
>>>>>>>>
>>>>>>>> I've got the feeling that all this submodule crap is constantly causing
>>>>>>>> pain ... do we really need this? Can't we find another solution instead?
>>>>>>>> Or at least stop modifying files automatically in the $SRC_PATH ?
>>>>>>>
>>>>>>> Also yesterday on IRC:
>>>>>>>
>>>>>>> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
>>>>>>> it. I am getting this:
>>>>>>>
>>>>>>> ./configure --static && make && sudo make install
>>>>>>>  CC      ui/input-keymap.o
>>>>>>> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
>>>>>>> No such file or directory
>>>>>>
>>>>>> I had a pull request merged yesterday later afternoon which possibly
>>>>>> would address that problem, though hard hard to say for certain.
>>>>>
>>>>> wow, already? :(
>>>>>
>>>>> I still wonder why do not we checkout submodules into the build directory
>>>>> and why .git-submodule-status is not there too...
>>>>
>>>> That simply isn't the way submodules work, they are inherently part of
>>>> the source tree, and the status file reflects that too.
>>>
>>> Sorry, I am missing the point here. What precisely does prevent us from
>>> checking out the required modules to the build directory and build them
>>> there? git provides a submodule repository url and sha1 for the current
>>> qemu branch.
>>
>> The build directory should never contain any of your version controlled
>> source, as that will get irretrievably lost when the build dir is purged.
> 
> I am not suggesting editing submodule sources from the build directory, I
> am suggesting not building them from the source directory, in order to keep
> them in sync, Makefile can rsync or "git push" them to the build directory.
> Yeah, fairly ugly but still more correct than the current solution.
> 
> And again, .git-submodule-status is not a source file, what is it doing in
> $SRC_PATH?
> 
> Now I cannot compile qemu pretty much 50% of time because of that
> enhancement. In the today's episode:
> 
> ssh aikhostos2 cd /home/aik/pbuild/qemu-aikhostos2-ppc64/ ;
> /home/aik/p/qemu/configure --target-list=ppc64-softmmu
> --source-path=/home/aik/p/qemu/ --enable-debug --enable-debug-info
> --disable-werror --disable-git-update --enable-trace-backend=log
> 
> All good. One detail:
> capstone          git
> 
> ssh aikhostos2 make -C /home/aik/pbuild/qemu-aikhostos2-ppc64/ -j24
> 
> make: Entering directory `/home/aik/pbuild/qemu-aikhostos2-ppc64'
>   GEN     ppc64-softmmu/config-devices.mak.tmp
> mkdir -p dtc/libfdt
> mkdir -p dtc/tests
>   GEN     config-host.h
>   GEN     qemu-options.def
> make[1]: *** No rule to make target
> `/home/aik/pbuild/qemu-aikhostos2-ppc64/capstone/libcapstone.a'.  Stop.
> make: *** [subdir-capstone] Error 2
> make: *** Waiting for unfinished jobs....
>   GEN     ppc64-softmmu/config-devices.mak
> 
> 
> How, why, where are all these nice warning messages? Ah, there they are
> (added 'set -x' to the script):
> 
> + substat=.git-submodule-status
> + command=status
> + shift
> + maybe_modules='ui/keycodemapdb dtc capstone'
> + test -z git
> + modules=
> + for m in '$maybe_modules'
> + git submodule status ui/keycodemapdb
> + test 128 = 0
> + echo 'warn: ignoring non-existent submodule ui/keycodemapdb'
> + for m in '$maybe_modules'
> + git submodule status dtc
> + test 128 = 0:m
> + echo 'warn: ignoring non-existent submodule dtc'
> + for m in '$maybe_modules'
> + git submodule status capstone
> + test 128 = 0
> + echo 'warn: ignoring non-existent submodule capstone'
> + test -n 'ui/keycodemapdb dtc capstone'
> + test -e .git
> + case "$command" in
> + test -z 'ui/keycodemapdb dtc capstone'
> + test -f .git-submodule-status
> + exit 1
> 
> 
> I assume that 'warn' is not printed because of Makefile's 'quiet-command'.
> 
> And this fails because on the server the source directory is created with
> 'git worktree' which points to bare git repo outside of the source tree and
> the git on that build machine has no idea about worktrees (the repo is
> visible to the build machine).
> 
> [vpl1 ~]$ ssh -x aikhostos2 'cd p/qemu && pwd && git --version && ls -d
> ../qemu.git && git submodule status'
> /home/aik/p/qemu
> git version 1.8.3.1
> ../qemu.git
> fatal: Not a git repository: /home/aik/p/qemu.git/worktrees/qemu
> 
> 
> Again - ok, when I know what is the problem, I know how to fix it - simply
> move .git away when running ./configure and make (as it also runs
> ./configure some time) - but the messages... Firstly ./configure decides
> that 'capstone' comes from 'git', then it simply fails with no indication
> why it went wrong.
> 
> Ha. I just found another workaround - I can do ./configure --with-git=false
> and silently disable git updates. Pretty much pointing a gun to my leg,
> ready to shoot.
> 
> We could just check git submodule status every build and print a warning if
> it is out of sync but continue. This way I would look at the submodules
> list and decide what to do.


I assume that no response to this means my proposal is somehow worse that
what we got now, correct? Thanks.



-- 
Alexey