git gc seems to break --symbolic-full-name

git gc seems to break --symbolic-full-name

[Stas Sergeev]

Hello.

I do the following:

$ git rev-parse --symbolic-full-name devel
refs/heads/devel
$ ls -l .git/`git rev-parse --symbolic-full-name devel`
-rw-rw-r-- 1 stas stas 41 июл 21 15:05 .git/refs/heads/devel

This is fine. But after git gc:

$ git rev-parse --symbolic-full-name devel
refs/heads/devel
$ LC_ALL=C ls -l .git/`git rev-parse --symbolic-full-name devel`
ls: cannot access '.git/refs/heads/devel': No such file or directory

This is because after git gc there is nothing in .git/refs/heads:

$ LC_ALL=C ls -l .git/refs/heads/
total 8
drwxrwxr-x 2 stas stas 4096 Jul 31  2013 feature
drwxrwxr-x 2 stas stas 4096 Dec 23  2016 pr

I prepend the path with .git above just as an
example: the same happens with rev-parse --git-path.

So I wonder if it is an expected behaviour for
--symbolic-full-name to return the invalid pathes?
If so, how can I get the path to the needed head file?
I use that for dependency tracking.

Re: git gc seems to break --symbolic-full-name

[Junio C Hamano]

Stas Sergeev <stsp@list.ru> writes:

> I do the following:
>
> $ git rev-parse --symbolic-full-name devel
> refs/heads/devel
> $ ls -l .git/`git rev-parse --symbolic-full-name devel`
> -rw-rw-r-- 1 stas stas 41 июл 21 15:05 .git/refs/heads/devel
>
> This is fine. But after git gc:
>
> $ git rev-parse --symbolic-full-name devel
> refs/heads/devel
> $ LC_ALL=C ls -l .git/`git rev-parse --symbolic-full-name devel`
> ls: cannot access '.git/refs/heads/devel': No such file or directory

This is expected, and in the modern world (like, after year 2007),
a refname "refs/heads/foo" does *not* mean that there is a file
with such a path under .git/ directory.  "rev-parse" does not return
any "path" in the filesystem sense.

See "git pack-refs --help", and depending on what you want to learn
about the ref in question, perhaps "git show-ref refs/heads/devel"
is what you want.

Re: git gc seems to break --symbolic-full-name

[Stas Sergeev]

21.07.2017 21:56, Junio C Hamano пишет:
> Stas Sergeev <stsp@list.ru> writes:
>
>> I do the following:
>>
>> $ git rev-parse --symbolic-full-name devel
>> refs/heads/devel
>> $ ls -l .git/`git rev-parse --symbolic-full-name devel`
>> -rw-rw-r-- 1 stas stas 41 июл 21 15:05 .git/refs/heads/devel
>>
>> This is fine. But after git gc:
>>
>> $ git rev-parse --symbolic-full-name devel
>> refs/heads/devel
>> $ LC_ALL=C ls -l .git/`git rev-parse --symbolic-full-name devel`
>> ls: cannot access '.git/refs/heads/devel': No such file or directory
> This is expected, and in the modern world (like, after year 2007),
> a refname "refs/heads/foo" does *not* mean that there is a file
> with such a path under .git/ directory.  "rev-parse" does not return
> any "path" in the filesystem sense.
>
> See "git pack-refs --help", and depending on what you want to learn
> about the ref in question, perhaps "git show-ref refs/heads/devel"
> is what you want.
I wanted some kind of file to use it as a
build dependency for the files that needs
to be re-built when the head changes.
This works very well besides git gc.
What other method can be used as simply
as that? git show-ref does not seem to be
giving this.

Re: git gc seems to break --symbolic-full-name

[Jacob Keller]

On Fri, Jul 21, 2017 at 12:03 PM, Stas Sergeev <stsp@list.ru> wrote:
> I wanted some kind of file to use it as a
> build dependency for the files that needs
> to be re-built when the head changes.
> This works very well besides git gc.
> What other method can be used as simply
> as that? git show-ref does not seem to be
> giving this.

There's no real way to do this, and even prior to 2007 when the file
always existed, there's no guarantee it's modification time is valid.

I'd suggest you have a phony rule which you always run, that checks
the ref, and sees if it's different from "last time" and then updates
a different file if that's the case. Then the build can depend on the
generated file, and you'd be able to figure it out.

What's the real goal for depending on when the ref changes?

Thanks,
Jake

Re: git gc seems to break --symbolic-full-name

[Stas Sergeev]

23.07.2017 11:40, Jacob Keller пишет:
> On Fri, Jul 21, 2017 at 12:03 PM, Stas Sergeev <stsp@list.ru> wrote:
>> I wanted some kind of file to use it as a
>> build dependency for the files that needs
>> to be re-built when the head changes.
>> This works very well besides git gc.
>> What other method can be used as simply
>> as that? git show-ref does not seem to be
>> giving this.
> There's no real way to do this, and even prior to 2007 when the file
> always existed, there's no guarantee it's modification time is valid.
>
> I'd suggest you have a phony rule which you always run, that checks
> the ref, and sees if it's different from "last time" and then updates
> a different file if that's the case. Then the build can depend on the
> generated file, and you'd be able to figure it out.
OK, thanks, that looks quite simple too.
I will have to create the file by hands that
I expected git to already have, but it appears
not.

> What's the real goal for depending on when the ref changes?
So that when users fill in the bug report, I can
see at what revision have the bug happened. :)
While seemingly "just a debugging sugar", the
hard experience shows this to be exceptionally
useful.
I think even linux kernel does something like
this, and solves that task the hard way. For
example I can see a script at scripts/setlocalversion
whose output seems to go to
include/config/kernel.release and a lot of
logic in the toplevel makefile about this.
So not liking the fact that every project solves
this differently, I was trying to get the solution
directly from git. But I'll try otherwise.

Re: git gc seems to break --symbolic-full-name

[Jacob Keller]

On Sun, Jul 23, 2017 at 12:23 PM, Stas Sergeev <stsp@list.ru> wrote:
> 23.07.2017 11:40, Jacob Keller пишет:
>>
>> On Fri, Jul 21, 2017 at 12:03 PM, Stas Sergeev <stsp@list.ru> wrote:
>>>
>>> I wanted some kind of file to use it as a
>>> build dependency for the files that needs
>>> to be re-built when the head changes.
>>> This works very well besides git gc.
>>> What other method can be used as simply
>>> as that? git show-ref does not seem to be
>>> giving this.
>>
>> There's no real way to do this, and even prior to 2007 when the file
>> always existed, there's no guarantee it's modification time is valid.
>>
>> I'd suggest you have a phony rule which you always run, that checks
>> the ref, and sees if it's different from "last time" and then updates
>> a different file if that's the case. Then the build can depend on the
>> generated file, and you'd be able to figure it out.
>
> OK, thanks, that looks quite simple too.
> I will have to create the file by hands that
> I expected git to already have, but it appears
> not.
>
>> What's the real goal for depending on when the ref changes?
>
> So that when users fill in the bug report, I can
> see at what revision have the bug happened. :)
> While seemingly "just a debugging sugar", the
> hard experience shows this to be exceptionally
> useful.
> I think even linux kernel does something like
> this, and solves that task the hard way. For
> example I can see a script at scripts/setlocalversion
> whose output seems to go to
> include/config/kernel.release and a lot of
> logic in the toplevel makefile about this.
> So not liking the fact that every project solves
> this differently, I was trying to get the solution
> directly from git. But I'll try otherwise.

generally, I'd suggest using "git describe" to output a version based
on tag, and as part of your build system set that in some sort of
--version output of some kind.

Thanks,
Jake

Re: git gc seems to break --symbolic-full-name

[Stas Sergeev]

24.07.2017 07:02, Jacob Keller пишет:
> generally, I'd suggest using "git describe" to output a version based
> on tag, and as part of your build system set that in some sort of
> --version output of some kind.
I am not sure I understand that suggestion.
I can use 'git describe' (and that seems to be
what linux kernel does too), but I don't see
how can I get away without a temporary file,
maually comparing current 'git describe' with
the one stored, and updating that file in case
of a mismatch.
Could you please clarify? Maybe I am missing
some makefile trick which you assume I am
aware about. :)

Re: git gc seems to break --symbolic-full-name

[Stas Sergeev]

24.07.2017 07:02, Jacob Keller пишет:
> generally, I'd suggest using "git describe" to output a version based
> on tag, and as part of your build system set that in some sort of
> --version output of some kind.
I came to the following solution which
looks quite simple (avoids comparing the
output of git describe):
git log -1 --format=%cd --date=rfc | xargs -I {} touch --date={} $TSTAMP

The care must be taken to put the timestamp
file as a pre-requisite of the .LOW_RESOLUTION_TIME
target, and the build seems to work properly.
This still smells hackish, but this time I blame
make for an inability to specify the timestamps
explicitly. :)

Re: git gc seems to break --symbolic-full-name

[Jacob Keller]

On Sun, Jul 23, 2017 at 12:23 PM, Stas Sergeev <stsp@list.ru> wrote:
> 23.07.2017 11:40, Jacob Keller пишет:
>>
>> On Fri, Jul 21, 2017 at 12:03 PM, Stas Sergeev <stsp@list.ru> wrote:
>>>
>>> I wanted some kind of file to use it as a
>>> build dependency for the files that needs
>>> to be re-built when the head changes.
>>> This works very well besides git gc.
>>> What other method can be used as simply
>>> as that? git show-ref does not seem to be
>>> giving this.
>>
>> There's no real way to do this, and even prior to 2007 when the file
>> always existed, there's no guarantee it's modification time is valid.
>>
>> I'd suggest you have a phony rule which you always run, that checks
>> the ref, and sees if it's different from "last time" and then updates
>> a different file if that's the case. Then the build can depend on the
>> generated file, and you'd be able to figure it out.
>
> OK, thanks, that looks quite simple too.
> I will have to create the file by hands that
> I expected git to already have, but it appears
> not.
>
>> What's the real goal for depending on when the ref changes?
>
> So that when users fill in the bug report, I can
> see at what revision have the bug happened. :)
> While seemingly "just a debugging sugar", the
> hard experience shows this to be exceptionally
> useful.
> I think even linux kernel does something like
> this, and solves that task the hard way. For
> example I can see a script at scripts/setlocalversion
> whose output seems to go to
> include/config/kernel.release and a lot of
> logic in the toplevel makefile about this.
> So not liking the fact that every project solves
> this differently, I was trying to get the solution
> directly from git. But I'll try otherwise.

If your goal is to make it so users filling out bug reports have a
version, then using git descrsibe and making that be part of your
version (based off your tags, and commits) is how pretty much every
other project I've worked on does this.

I really don't think that's your goal here, given you're doing things
in make with timestamps and builds, so I guess I misunderstood your
answer?

Thanks,
Jake

Re: git gc seems to break --symbolic-full-name

[Stas Sergeev]

26.07.2017 03:36, Jacob Keller пишет:
> If your goal is to make it so users filling out bug reports have a
> version, then using git descrsibe and making that be part of your
> version (based off your tags, and commits) is how pretty much every
> other project I've worked on does this.
>
> I really don't think that's your goal here, given you're doing things
> in make with timestamps and builds, so I guess I misunderstood your
> answer?
There are 2 different things:
1. put git describe output into some header file
2. make things to rebuild with every new commit

So I actually stuck at solving 2, because 1 is trivial.
My original solution for 2 was to "depend" on
refs/heads/*. This worked besides git gc, but had
a lot of troubles with worktrees. And this time I
switched to the "touch tmpfile" trick with the date
taken from git log. This requires .LOW_RESOLUTION_TIME
in makefile, so probably not the best solution again,
but should hopefully be more future-proof than the
previous one.

Re: git gc seems to break --symbolic-full-name

[Jeff King]

On Wed, Jul 26, 2017 at 11:40:46AM +0300, Stas Sergeev wrote:

> 26.07.2017 03:36, Jacob Keller пишет:
> > If your goal is to make it so users filling out bug reports have a
> > version, then using git descrsibe and making that be part of your
> > version (based off your tags, and commits) is how pretty much every
> > other project I've worked on does this.
> > 
> > I really don't think that's your goal here, given you're doing things
> > in make with timestamps and builds, so I guess I misunderstood your
> > answer?
> There are 2 different things:
> 1. put git describe output into some header file
> 2. make things to rebuild with every new commit
> 
> So I actually stuck at solving 2, because 1 is trivial.
> My original solution for 2 was to "depend" on
> refs/heads/*. This worked besides git gc, but had
> a lot of troubles with worktrees. And this time I
> switched to the "touch tmpfile" trick with the date
> taken from git log. This requires .LOW_RESOLUTION_TIME
> in makefile, so probably not the best solution again,
> but should hopefully be more future-proof than the
> previous one.

In git.git we do something like:

-- >8 --
other: version
	cat $< >$@

.PHONY: FORCE
version: FORCE
	@git rev-parse HEAD >$@+
	@if cmp $@+ $@ >/dev/null 2>&1; then rm $@+; else mv $@+ $@; fi
-- >8 --

The "version" commands run always, but they only update the file if
there's a change. At least GNU make is smart enough to know that if
"version" was not updated, then "other" does not need to be re-built.
I don't know if all makes would so, though.

-Peff

Re: git gc seems to break --symbolic-full-name

[Stas Sergeev]

26.07.2017 16:23, Jeff King пишет:
> In git.git we do something like:
>
> -- >8 --
> other: version
> 	cat $< >$@
>
> .PHONY: FORCE
> version: FORCE
> 	@git rev-parse HEAD >$@+
> 	@if cmp $@+ $@ >/dev/null 2>&1; then rm $@+; else mv $@+ $@; fi
> -- >8 --
Yes, thats a nice recipe that I would be using
if not for the fact that I already switched to
"touch", which requires 1 fewer tmp file and
no comparison.
But I'll keep this in mind if something wrong
happens with my current solution, thanks!
I wonder if git can provide some helper script
for other projects to solve this in a same way.