Temporary git files for the gitdir created on a separate drive in workdir

Temporary git files for the gitdir created on a separate drive in workdir

[Hultqvist]

I'm having two types of git repos in separate drives, in this case G:
is a Google Drive File Stream meaning a slow synchronized storage.

First repo:
Workdir: G:\Test1
Gitdir: C:\Test1.git
G:\Test1\.git containing "gitdir: C:\Test1.git"

Second repo:
Workdir: G:\Test2
Gitdir: G:\Test2\.git

In the first repo I see a lot of "master", "config" and "index" files
being created in the root of the workdir.
The nature of the slow sync causes these files to duplicate - not gits fault.

In the second repo no artefacts are noticed.

I understand that the files are first created "somewhere else" to make
an atomic move later.
In the first repo, the nature of being two separate drives appear to
cause the residue.
In the second repo it works right.

Considering that the gitdir could be located on a different drive than
the workdir wouldn't it make more sense to create the temporary files
in a subdirectory inside the gitdir rather tan in the workdir?

Are there any flags or environment variables to control where
temporary files are created?

Re: Temporary git files for the gitdir created on a separate drive in workdir

[Junio C Hamano]

Hultqvist <hultqvist@silentorbit.com> writes:

> Considering that the gitdir could be located on a different drive than
> the workdir wouldn't it make more sense to create the temporary files
> in a subdirectory inside the gitdir rather tan in the workdir?

I do not think we intend to create temporary files, whose final
destination is somewhere under $GIT_DIR/, in any working tree;
rather, I think we try to create them inside $GIT_DIR (or possibly
if the destination is a file in a subdirectory of $GIT_DIR, then in
the same subdirectory).  What you are seeing definitely smells like
a bug in the worktree code, perhaps getting confused by the fact
that the full path to these places look "unusual" by starting with a
single alphabet followed by a colon (IOW, this may manifest only in
Windows port).

Re: Temporary git files for the gitdir created on a separate drive in workdir

[Duy Nguyen]

On Fri, Sep 7, 2018 at 6:48 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Hultqvist <hultqvist@silentorbit.com> writes:
>
> > Considering that the gitdir could be located on a different drive than
> > the workdir wouldn't it make more sense to create the temporary files
> > in a subdirectory inside the gitdir rather tan in the workdir?
>
> I do not think we intend to create temporary files, whose final
> destination is somewhere under $GIT_DIR/, in any working tree;
> rather, I think we try to create them inside $GIT_DIR (or possibly
> if the destination is a file in a subdirectory of $GIT_DIR, then in
> the same subdirectory).  What you are seeing definitely smells like
> a bug in the worktree code, perhaps getting confused by the fact
> that the full path to these places look "unusual" by starting with a
> single alphabet followed by a colon (IOW, this may manifest only in
> Windows port).

I agree. Auditing the setup code did not reveal anything though. Our
code should recognize these unusual Windows paths as absolute and
while I spotted an incorrect use of '/' (instead of is_dir_sep) it
does not explain the problem here.

Hultqvist, if you set environment variable GIT_TRACE_SETUP to 1 and
run "git status" in G:\Test1, what does it say?
-- 
Duy

Re: Temporary git files for the gitdir created on a separate drive in workdir

[Hultqvist]

The bash commands are using a git and bash bundle that was installed
in parallel with gitextensions(a gui for git)

G:\Min enhet> set GIT_TRACE_SETUP=1
G:\Min enhet> git st
10:40:28.881927 trace.c:318             setup: git_dir:
C:/Users/hultqvist/Drive.git
10:40:28.881927 trace.c:319             setup: git_common_dir:
C:/Users/hultqvist/Drive.git
10:40:28.881927 trace.c:320             setup: worktree: G:/Min enhet
10:40:28.881927 trace.c:321             setup: cwd: G:/Min enhet
10:40:28.881927 trace.c:322             setup: prefix: (null)
10:40:28.882930 chdir-notify.c:67       setup: chdir from 'G:/Min
enhet' to 'G:/Min enhet'
On branch master
Your branch is up to date with 'nas/master'.

nothing to commit, working tree clean

$ cat .git
gitdir: C:\Users\hultqvist\Drive.git

$cat C:/Users/hultqvist/Drive.git/config
[core]
        repositoryformatversion = 0
        filemode = false
        bare = false
        logallrefupdates = true
        ignorecase = true
        autocrlf = false
...

G:\Min enhet> git version
$ git version
git version 2.18.0.windows.1




Den lör 8 sep. 2018 kl 08:08 skrev Duy Nguyen <pclouds@gmail.com>:
>
> On Fri, Sep 7, 2018 at 6:48 PM Junio C Hamano <gitster@pobox.com> wrote:
> >
> > Hultqvist <hultqvist@silentorbit.com> writes:
> >
> > > Considering that the gitdir could be located on a different drive than
> > > the workdir wouldn't it make more sense to create the temporary files
> > > in a subdirectory inside the gitdir rather tan in the workdir?
> >
> > I do not think we intend to create temporary files, whose final
> > destination is somewhere under $GIT_DIR/, in any working tree;
> > rather, I think we try to create them inside $GIT_DIR (or possibly
> > if the destination is a file in a subdirectory of $GIT_DIR, then in
> > the same subdirectory).  What you are seeing definitely smells like
> > a bug in the worktree code, perhaps getting confused by the fact
> > that the full path to these places look "unusual" by starting with a
> > single alphabet followed by a colon (IOW, this may manifest only in
> > Windows port).
>
> I agree. Auditing the setup code did not reveal anything though. Our
> code should recognize these unusual Windows paths as absolute and
> while I spotted an incorrect use of '/' (instead of is_dir_sep) it
> does not explain the problem here.
>
> Hultqvist, if you set environment variable GIT_TRACE_SETUP to 1 and
> run "git status" in G:\Test1, what does it say?
> --
> Duy

Re: Temporary git files for the gitdir created on a separate drive in workdir

[Duy Nguyen]

On Sat, Sep 8, 2018 at 11:28 AM Hultqvist <hultqvist@silentorbit.com> wrote:
>
> The bash commands are using a git and bash bundle that was installed
> in parallel with gitextensions(a gui for git)
>
> G:\Min enhet> set GIT_TRACE_SETUP=1
> G:\Min enhet> git st
> 10:40:28.881927 trace.c:318             setup: git_dir:
> C:/Users/hultqvist/Drive.git
> 10:40:28.881927 trace.c:319             setup: git_common_dir:
> C:/Users/hultqvist/Drive.git
> 10:40:28.881927 trace.c:320             setup: worktree: G:/Min enhet
> 10:40:28.881927 trace.c:321             setup: cwd: G:/Min enhet
> 10:40:28.881927 trace.c:322             setup: prefix: (null)
> 10:40:28.882930 chdir-notify.c:67       setup: chdir from 'G:/Min
> enhet' to 'G:/Min enhet'

Unfortunately this looks good. Whenever those files 'index',
'config'... are created, they should always be created in
C:\Users\hultqvist\Drive.git, not G:\Min enhet, including their
temporary versions. I don't know if there are any more changes on the
windows fork that could affect this though, I only checked git.git.
-- 
Duy

Re: Temporary git files for the gitdir created on a separate drive in workdir

[Duy Nguyen]

On Sat, Sep 8, 2018 at 3:09 PM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Sat, Sep 8, 2018 at 11:28 AM Hultqvist <hultqvist@silentorbit.com> wrote:
> >
> > The bash commands are using a git and bash bundle that was installed
> > in parallel with gitextensions(a gui for git)
> >
> > G:\Min enhet> set GIT_TRACE_SETUP=1
> > G:\Min enhet> git st
> > 10:40:28.881927 trace.c:318             setup: git_dir:
> > C:/Users/hultqvist/Drive.git
> > 10:40:28.881927 trace.c:319             setup: git_common_dir:
> > C:/Users/hultqvist/Drive.git
> > 10:40:28.881927 trace.c:320             setup: worktree: G:/Min enhet
> > 10:40:28.881927 trace.c:321             setup: cwd: G:/Min enhet
> > 10:40:28.881927 trace.c:322             setup: prefix: (null)
> > 10:40:28.882930 chdir-notify.c:67       setup: chdir from 'G:/Min
> > enhet' to 'G:/Min enhet'
>
> Unfortunately this looks good. Whenever those files 'index',
> 'config'... are created, they should always be created in
> C:\Users\hultqvist\Drive.git, not G:\Min enhet, including their
> temporary versions. I don't know if there are any more changes on the
> windows fork that could affect this though, I only checked git.git.

BTW do you notice these files showing up after any particular command
or they're always there after cloning? Perhaps some command got the
".git" directory discovery wrong and assumed $GIT_DIR=$GIT_WORK_TREE.
We have a much bigger problem then.
-- 
Duy

Re: Temporary git files for the gitdir created on a separate drive in workdir

[Hultqvist]

Since this thread started I haven't seen a single file mentioned being created,
Usually they appear during work days when there is more activity.
I've never seen the files created directly, only a larger amount of
them once in a while.

I will run process monitor and get back once I find out more.
Den lör 8 sep. 2018 kl 15:44 skrev Duy Nguyen <pclouds@gmail.com>:
>
> On Sat, Sep 8, 2018 at 3:09 PM Duy Nguyen <pclouds@gmail.com> wrote:
> >
> > On Sat, Sep 8, 2018 at 11:28 AM Hultqvist <hultqvist@silentorbit.com> wrote:
> > >
> > > The bash commands are using a git and bash bundle that was installed
> > > in parallel with gitextensions(a gui for git)
> > >
> > > G:\Min enhet> set GIT_TRACE_SETUP=1
> > > G:\Min enhet> git st
> > > 10:40:28.881927 trace.c:318             setup: git_dir:
> > > C:/Users/hultqvist/Drive.git
> > > 10:40:28.881927 trace.c:319             setup: git_common_dir:
> > > C:/Users/hultqvist/Drive.git
> > > 10:40:28.881927 trace.c:320             setup: worktree: G:/Min enhet
> > > 10:40:28.881927 trace.c:321             setup: cwd: G:/Min enhet
> > > 10:40:28.881927 trace.c:322             setup: prefix: (null)
> > > 10:40:28.882930 chdir-notify.c:67       setup: chdir from 'G:/Min
> > > enhet' to 'G:/Min enhet'
> >
> > Unfortunately this looks good. Whenever those files 'index',
> > 'config'... are created, they should always be created in
> > C:\Users\hultqvist\Drive.git, not G:\Min enhet, including their
> > temporary versions. I don't know if there are any more changes on the
> > windows fork that could affect this though, I only checked git.git.
>
> BTW do you notice these files showing up after any particular command
> or they're always there after cloning? Perhaps some command got the
> ".git" directory discovery wrong and assumed $GIT_DIR=$GIT_WORK_TREE.
> We have a much bigger problem then.
> --
> Duy

Re: Temporary git files for the gitdir created on a separate drive in workdir

[Hultqvist]

Sending again without HTML

Den mån 10 sep. 2018 kl 12:28 skrev Hultqvist <hultqvist@silentorbit.com>:
>
> First I need to correct my previous observations.
>
> Today there appeared new set of config files in the root.
> I looked into a few of them and found that their content doesn't match that of the repo at "G:/Min enhet".
> Instead separate files had content from separate git repos within the G drive.
> These repos are not like the one we're discussed previously, they are completely within G: using a classical .git directory.
>
> I guess git is creating the temporary files as close as possible to the root, since "G:\" can't be written to, only "G:\Min enhet". and then copy them to the final destination which in this case is the same drive.
> If so then we can't get away from the duplicate files, the duplicates themselves are the fault of Google Drive File Stream.
> If git would create the config/master/index files within the .git dir to start with that would hide the clutter from the root but not prevent the duplicates.
> Is this observation correct, that git creates temporary files closer to the root than inside the .git directory or a sub directory thereof?
>
> I haven't experiences any issues working with these repos themselves.
> One risk could be that the final "index" is an older version than "index (45)".
> I'm certain this haven't been the case since I push every commit to a remote repo that doesn't accept changes in history.
>
> I understand that master and index needs to be updated with regular use.
> Does config need the same amount of updates?
> I've considered that file to only change when I make explicit changes, not during regular use(push, commit, status).
>
> Thanks for the help looking into this for me.
>
>
> Den sön 9 sep. 2018 kl 17:30 skrev Hultqvist <hultqvist@silentorbit.com>:
>>
>> Since this thread started I haven't seen a single file mentioned being created,
>> Usually they appear during work days when there is more activity.
>> I've never seen the files created directly, only a larger amount of
>> them once in a while.
>>
>> I will run process monitor and get back once I find out more.
>> Den lör 8 sep. 2018 kl 15:44 skrev Duy Nguyen <pclouds@gmail.com>:
>> >
>> > On Sat, Sep 8, 2018 at 3:09 PM Duy Nguyen <pclouds@gmail.com> wrote:
>> > >
>> > > On Sat, Sep 8, 2018 at 11:28 AM Hultqvist <hultqvist@silentorbit.com> wrote:
>> > > >
>> > > > The bash commands are using a git and bash bundle that was installed
>> > > > in parallel with gitextensions(a gui for git)
>> > > >
>> > > > G:\Min enhet> set GIT_TRACE_SETUP=1
>> > > > G:\Min enhet> git st
>> > > > 10:40:28.881927 trace.c:318             setup: git_dir:
>> > > > C:/Users/hultqvist/Drive.git
>> > > > 10:40:28.881927 trace.c:319             setup: git_common_dir:
>> > > > C:/Users/hultqvist/Drive.git
>> > > > 10:40:28.881927 trace.c:320             setup: worktree: G:/Min enhet
>> > > > 10:40:28.881927 trace.c:321             setup: cwd: G:/Min enhet
>> > > > 10:40:28.881927 trace.c:322             setup: prefix: (null)
>> > > > 10:40:28.882930 chdir-notify.c:67       setup: chdir from 'G:/Min
>> > > > enhet' to 'G:/Min enhet'
>> > >
>> > > Unfortunately this looks good. Whenever those files 'index',
>> > > 'config'... are created, they should always be created in
>> > > C:\Users\hultqvist\Drive.git, not G:\Min enhet, including their
>> > > temporary versions. I don't know if there are any more changes on the
>> > > windows fork that could affect this though, I only checked git.git.
>> >
>> > BTW do you notice these files showing up after any particular command
>> > or they're always there after cloning? Perhaps some command got the
>> > ".git" directory discovery wrong and assumed $GIT_DIR=$GIT_WORK_TREE.
>> > We have a much bigger problem then.
>> > --
>> > Duy

Re: Temporary git files for the gitdir created on a separate drive in workdir

[Duy Nguyen]

On Mon, Sep 10, 2018 at 12:27 PM Hultqvist <hultqvist@silentorbit.com> wrote:
>
> Sending again without HTML
>
> Den mån 10 sep. 2018 kl 12:28 skrev Hultqvist <hultqvist@silentorbit.com>:
> >
> > First I need to correct my previous observations.

Please don't top-post.

> > Today there appeared new set of config files in the root.
> > I looked into a few of them and found that their content doesn't match that of the repo at "G:/Min enhet".
> > Instead separate files had content from separate git repos within the G drive.
> > These repos are not like the one we're discussed previously, they are completely within G: using a classical .git directory.
> >
> > I guess git is creating the temporary files as close as possible to the root, since "G:\" can't be written to, only "G:\Min enhet". and then copy them to the final destination which in this case is the same drive.

No. Those files should always be created inside the ".git" directory,
wherever it is. Failing to creating a file in there is usually a
serious error and the command will abort. Unless there is a bug
lurking around of course, but I can't nail it down with just code
audit.

Since the content of those files does not look like from "G:\Min
enhet" repo, do you know which repo they belong to (and where those
repo and worktree are)? I ask because if these are submodules of
"G:\Min enhet" for example, then we need to head another direction. Or
if they are completely unrelated to "G:\Min enhet", oh boy...
-- 
Duy