[PATCH] config.mak.uname: Cygwin: Use renames for creation

[PATCH] config.mak.uname: Cygwin: Use renames for creation

[Adam Dinwoodie]

When generating build options for Cygwin, enable
OBJECT_CREATION_USES_RENAMES.  This is necessary to use Git on Windows
shared directories, and is already enabled for the MinGW and plain
Windows builds.

This problem was reported on the Cygwin mailing list at
https://cygwin.com/ml/cygwin/2015-08/msg00102.html (amongst others) and
is being applied as a manual patch to the Cygwin builds until the patch
is taken here.

Reported-by: Peter Rosin <peda@lysator.liu.se>
Signed-off-by: Adam Dinwoodie <adam@dinwoodie.org>
---
 config.mak.uname | 1 +
 1 file changed, 1 insertion(+)

diff --git a/config.mak.uname b/config.mak.uname
index 943c439..be5cbec 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -187,6 +187,7 @@ ifeq ($(uname_O),Cygwin)
 	X = .exe
 	UNRELIABLE_FSTAT = UnfortunatelyYes
 	SPARSE_FLAGS = -isystem /usr/include/w32api -Wno-one-bit-signed-bitfield
+	OBJECT_CREATION_USES_RENAMES = UnfortunatelyNeedsTo
 endif
 ifeq ($(uname_S),FreeBSD)
 	NEEDS_LIBICONV = YesPlease
-- 
2.4.5

Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation

[Mark Levedahl]

On 08/07/2015 04:30 PM, Adam Dinwoodie wrote:
> When generating build options for Cygwin, enable
> OBJECT_CREATION_USES_RENAMES.  This is necessary to use Git on Windows
> shared directories, and is already enabled for the MinGW and plain
> Windows builds.
>
> This problem was reported on the Cygwin mailing list at
> https://cygwin.com/ml/cygwin/2015-08/msg00102.html (amongst others) and
> is being applied as a manual patch to the Cygwin builds until the patch
> is taken here.
>
> Reported-by: Peter Rosin <peda@lysator.liu.se>
> Signed-off-by: Adam Dinwoodie <adam@dinwoodie.org>
> ---
>   config.mak.uname | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/config.mak.uname b/config.mak.uname
> index 943c439..be5cbec 100644
> --- a/config.mak.uname
> +++ b/config.mak.uname
> @@ -187,6 +187,7 @@ ifeq ($(uname_O),Cygwin)
>   	X = .exe
>   	UNRELIABLE_FSTAT = UnfortunatelyYes
>   	SPARSE_FLAGS = -isystem /usr/include/w32api -Wno-one-bit-signed-bitfield
> +	OBJECT_CREATION_USES_RENAMES = UnfortunatelyNeedsTo
>   endif
>   ifeq ($(uname_S),FreeBSD)
>   	NEEDS_LIBICONV = YesPlease
I've been supporting use of git on cygwin since about 2008, this issue 
has never risen that I know. Whatever issue is being patched around 
here, if truly repeatable, should be handled by the cygwin dll as that 
code is focused on providing full linux compatibility. If git on linux 
does need this patch, git on cygwin should not, either. So, I vote 
against this.

Mark

Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

On Sat, Aug 08, 2015 at 04:47:46PM -0400, Mark Levedahl wrote:
> On 08/07/2015 04:30 PM, Adam Dinwoodie wrote:
> >When generating build options for Cygwin, enable
> >OBJECT_CREATION_USES_RENAMES.  This is necessary to use Git on Windows
> >shared directories, and is already enabled for the MinGW and plain
> >Windows builds.
> I've been supporting use of git on cygwin since about 2008, this issue has
> never risen that I know. Whatever issue is being patched around here, if
> truly repeatable, should be handled by the cygwin dll as that code is
> focused on providing full linux compatibility. If git on linux does need
> this patch, git on cygwin should not, either. So, I vote against this.

We've gotten a lot of users on the list who ask why their Git
directories on shared drives aren't working (or are broken in some way).
Since I don't use Windows, let me ask: does the Cygwin DLL handle
link(2) properly on shared drives, and if not, would this patch help it
do so?  I can imagine that perhaps SMB doesn't support the necessary
operations to make a POSIX link(2) work properly.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation

[Adam Dinwoodie]

On Sat, Aug 08, 2015 at 09:06:28PM +0000, brian m. carlson wrote:
> On Sat, Aug 08, 2015 at 04:47:46PM -0400, Mark Levedahl wrote:
> > On 08/07/2015 04:30 PM, Adam Dinwoodie wrote:
> > >When generating build options for Cygwin, enable
> > >OBJECT_CREATION_USES_RENAMES.  This is necessary to use Git on Windows
> > >shared directories, and is already enabled for the MinGW and plain
> > >Windows builds.
> >
> > I've been supporting use of git on cygwin since about 2008, this issue has
> > never risen that I know. Whatever issue is being patched around here, if
> > truly repeatable, should be handled by the cygwin dll as that code is
> > focused on providing full linux compatibility. If git on linux does need
> > this patch, git on cygwin should not, either. So, I vote against this.

There has been recent and historical discussion on the Cygwin mailing
list about this problem, as well as in other places like Stack Overflow.
I've put a link to one of those discussions on the Cygwin mailing list
in the original patch email.  I can also see some discussiions on this
list that seem related (search archives for "failed to read delta-pack
base object" and "Cygwin").

It may be the technically correct approach that the Cygwin library ought
to fix this, and indeed some improvements have been made in this area.
However given the limited interfaces that Windows offers here, a final
fix is very unlikely to come any time soon, so this patch is the
pragmatic solution.

I do not see any difference between the situation here and the situation
for MinGW, which is fundamentally a Cygwin fork, but which already has
this build option set for it in config.mak.uname.

> We've gotten a lot of users on the list who ask why their Git
> directories on shared drives aren't working (or are broken in some way).
> Since I don't use Windows, let me ask: does the Cygwin DLL handle
> link(2) properly on shared drives, and if not, would this patch help it
> do so?  I can imagine that perhaps SMB doesn't support the necessary
> operations to make a POSIX link(2) work properly.

I'd need to go back to the Cygwin list to get a definite answer, but as
I understand it, yes, this is is exactly the problem -- quoting Corinna,
one of the Cygwin project leads, "The MS NFS is not very reliable in
keeping up with changes to metadata."

We have verified that setting `core.createobject rename` resolves the
problem for people who are seeing it, which very strongly implies that
this build option would solve the problem similarly, but would fix it
for all users, not just those who spend enough time investigating the
problem to find that setting.

Adam

Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation

[Johannes Schindelin]

Hi Adam,

On 2015-08-09 04:01, Adam Dinwoodie wrote:

> I do not see any difference between the situation here and the situation
> for MinGW, which is fundamentally a Cygwin fork, but which already has
> this build option set for it in config.mak.uname.

This is incorrect. MinGW is distinctly *not* a Cygwin fork. MinGW means "Minimal GNU on Windows" and that in turn means that it provides an environment to build executables that purely use the Win32 API. Read: no POSIX emulation whatsoever. Most notably, MinGW programs cannot use fork(2); It is simply unavailable.

What you *probably* meant is that Git for Windows relies on MSys2 for its shell and Perl scripts, and that MSys2 in turn is a fork of Cygwin. That affects *only* the scripts, though; Git itself (as in `git.exe`) is still a pure MinGW program (and as a consequence, is quite a bit faster than Cygwin Git, at the price of certain quirks that Cygwin Git does not suffer).

>> We've gotten a lot of users on the list who ask why their Git
>> directories on shared drives aren't working (or are broken in some way).
>> Since I don't use Windows, let me ask: does the Cygwin DLL handle
>> link(2) properly on shared drives, and if not, would this patch help it
>> do so?  I can imagine that perhaps SMB doesn't support the necessary
>> operations to make a POSIX link(2) work properly.
> 
> I'd need to go back to the Cygwin list to get a definite answer, but as
> I understand it, yes, this is is exactly the problem -- quoting Corinna,
> one of the Cygwin project leads, "The MS NFS is not very reliable in
> keeping up with changes to metadata."
> 
> We have verified that setting `core.createobject rename` resolves the
> problem for people who are seeing it, which very strongly implies that
> this build option would solve the problem similarly, but would fix it
> for all users, not just those who spend enough time investigating the
> problem to find that setting.

>From my experience, it appears that providing Corinna Vinschen (or better put: the Cygwin developers in general) with a sound patch gets things fixed pretty timely.

And since `core.createObject = rename` seems to work around the problem, it should be possible to patch the Cygwin runtime accordingly. Sure, it will take a little investigation *what* code should be changed, and how, but the obvious benefit to *all* Cygwin applications should make that effort worth your while.

Please note that Cygwin's source code itself is in Git now, too: https://cygwin.com/git.html

Ciao,
Johannes

Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation

[Adam Dinwoodie]

On 09/08/2015 10:01, Johannes Schindelin wrote:
> On 2015-08-09 04:01, Adam Dinwoodie wrote:
>> I do not see any difference between the situation here and the situation
>> for MinGW, which is fundamentally a Cygwin fork, but which already has
>> this build option set for it in config.mak.uname.
> This is incorrect. MinGW is distinctly *not* a Cygwin fork. MinGW means "Minimal GNU on Windows" and that in turn means that it provides an environment to build executables that purely use the Win32 API. Read: no POSIX emulation whatsoever. Most notably, MinGW programs cannot use fork(2); It is simply unavailable.
>
> What you *probably* meant is that Git for Windows relies on MSys2 for its shell and Perl scripts, and that MSys2 in turn is a fork of Cygwin. That affects *only* the scripts, though; Git itself (as in `git.exe`) is still a pure MinGW program (and as a consequence, is quite a bit faster than Cygwin Git, at the price of certain quirks that Cygwin Git does not suffer).

You're quite right; I'd misread the MinGW page. Thank you for pointing 
that out :)

>>> We've gotten a lot of users on the list who ask why their Git
>>> directories on shared drives aren't working (or are broken in some way).
>>> Since I don't use Windows, let me ask: does the Cygwin DLL handle
>>> link(2) properly on shared drives, and if not, would this patch help it
>>> do so?  I can imagine that perhaps SMB doesn't support the necessary
>>> operations to make a POSIX link(2) work properly.
>> I'd need to go back to the Cygwin list to get a definite answer, but as
>> I understand it, yes, this is is exactly the problem -- quoting Corinna,
>> one of the Cygwin project leads, "The MS NFS is not very reliable in
>> keeping up with changes to metadata."
>>
>> We have verified that setting `core.createobject rename` resolves the
>> problem for people who are seeing it, which very strongly implies that
>> this build option would solve the problem similarly, but would fix it
>> for all users, not just those who spend enough time investigating the
>> problem to find that setting.
>  From my experience, it appears that providing Corinna Vinschen (or better put: the Cygwin developers in general) with a sound patch gets things fixed pretty timely.
>
> And since `core.createObject = rename` seems to work around the problem, it should be possible to patch the Cygwin runtime accordingly. Sure, it will take a little investigation *what* code should be changed, and how, but the obvious benefit to *all* Cygwin applications should make that effort worth your while.

Hmm. I'm not sure what a Cygwin fix would look like here. As I 
understand it, using link(2) will fail if the target exists, while using 
rename(2) will just clobber a target file.

If the desired goal is that Cygwin's link(2) acts like POSIX link(2) on 
network drives, I'm not convinced that's possible, at least not by 
emulating `core.createObject = rename` in the Cygwin library layer. The 
only implementations I can think of would introduce a window condition 
between checking for the target file's existence and then clobbering it. 
That'd mean the vast majority of the time the Cygwin command would work 
as expected, but there'd be race conditions where occasionally a file 
gets silently clobbered. Intermittent silent bugs seem much worse than 
the command reliably failing.

I think, given that, the current behaviour is preferable: calling link 
just errors out in this scenario, because there's no way to map it down 
to something that reliably acts like the POSIX call. It's then up to 
whatever program is trying to use the call to find an alternative way to 
achieve its desired result, and the consequences of that workaround are 
the responsibility of the calling application.

In short, as I understand it, this is exactly the scenario 
OBJECT_CREATION_USES_RENAMES is intended for.

Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation

[Junio C Hamano]

Adam Dinwoodie <adam@dinwoodie.org> writes:

> If the desired goal is that Cygwin's link(2) acts like POSIX link(2)
> on network drives, I'm not convinced that's possible, at least not by
> emulating `core.createObject = rename` in the Cygwin library
> layer.

The way core.createObject=rename makes things work is by avoiding
link(2) in the first place and using rename(2) instead.  You might
be able to emulate rename(2) of A to B by doing a link(2) of A to B
and then unlink(2) of A, but I do not think it is reasonable for the
system call emulation layer to detect a sequence of link followed by
unlink and use rename, i.e. emulationg the other way around.

So I suspect "fix in Cygwin" is a non-starter.

But in the end, I'd prefer the choice of the compiled-in default up
to the port maintainers.  You earlier said:

    This problem was reported on the Cygwin mailing list at
    https://cygwin.com/ml/cygwin/2015-08/msg00102.html (amongst others) and
    is being applied as a manual patch to the Cygwin builds until the patch
    is taken here.

so my preference is to see Cygwin continue to do so for a couple
release cycles of ours to make sure all Cygwin end-users are happy
and consider the flip of the default a good change for them, and
then the official Cygwin packager of Git sends a patch our way.

https://cygwin.com/ml/cygwin/2015-08/msg00102.html seems to indicate
that somebody called Adam Dinwoodie is wearing Git maintainer hat,
so perhaps you may be that "official Cygwin packager of Git" ;-)

I agree with everything you said in that message to Peter---the
patch should be included when you hear reports of `git config
core.createObject rename` helping more people.  After versions of
Cygwin Git package with such a change proves good, let's reduce the
workload of Cygwin Git maintainer by upstreaming that change to my
tree.  But not before.

Thanks.

Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation

[Adam Dinwoodie]

On Mon, Aug 10, 2015 at 12:08:29PM -0700, Junio C Hamano wrote:
> But in the end, I'd prefer the choice of the compiled-in default up
> to the port maintainers.  You earlier said:
> 
>     This problem was reported on the Cygwin mailing list at
>     https://cygwin.com/ml/cygwin/2015-08/msg00102.html (amongst others) and
>     is being applied as a manual patch to the Cygwin builds until the patch
>     is taken here.
> 
> so my preference is to see Cygwin continue to do so for a couple
> release cycles of ours to make sure all Cygwin end-users are happy
> and consider the flip of the default a good change for them, and
> then the official Cygwin packager of Git sends a patch our way.

Wait a few releases then resubmit assuming we've not had complaints from
Cygwin user.  Okay!

> https://cygwin.com/ml/cygwin/2015-08/msg00102.html seems to indicate
> that somebody called Adam Dinwoodie is wearing Git maintainer hat,
> so perhaps you may be that "official Cygwin packager of Git" ;-)

That would indeed be yours truly :)

> I agree with everything you said in that message to Peter---the
> patch should be included when you hear reports of `git config
> core.createObject rename` helping more people.  After versions of
> Cygwin Git package with such a change proves good, let's reduce the
> workload of Cygwin Git maintainer by upstreaming that change to my
> tree.  But not before.

Cool.  FWIW, the reason we've started applying this patch to the
downstream Cygwin builds is that I've now seen a number of reports
(primarily on Stack Overflow) of this problem where that config change
has fixed things.  I'd been holding off until I had those extra reports,
but now I have them I made the patch and figured I'd submit it upstream
at the same time.  As above, I'll wait a while until we're happy it's
stable, and resubmit at that point.

(On which note, I should probably submit the patch to the git-gui
Makefile we've had in our builds since back in v1.7.9, before I took
over the maintainership, since that one clearly is pretty stable...)

Thanks!

Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation

[Johannes Schindelin]

Hi Adam,

On 2015-08-09 19:05, Adam Dinwoodie wrote:
> On 09/08/2015 10:01, Johannes Schindelin wrote:
>> On 2015-08-09 04:01, Adam Dinwoodie wrote:
>>> 
>>>> We've gotten a lot of users on the list who ask why their Git
>>>> directories on shared drives aren't working (or are broken in some way).
>>>> Since I don't use Windows, let me ask: does the Cygwin DLL handle
>>>> link(2) properly on shared drives, and if not, would this patch help it
>>>> do so?  I can imagine that perhaps SMB doesn't support the necessary
>>>> operations to make a POSIX link(2) work properly.
>>>
>>> I'd need to go back to the Cygwin list to get a definite answer, but as
>>> I understand it, yes, this is is exactly the problem -- quoting Corinna,
>>> one of the Cygwin project leads, "The MS NFS is not very reliable in
>>> keeping up with changes to metadata."
>>>
>>> We have verified that setting `core.createobject rename` resolves the
>>> problem for people who are seeing it, which very strongly implies that
>>> this build option would solve the problem similarly, but would fix it
>>> for all users, not just those who spend enough time investigating the
>>> problem to find that setting.
>>
>>  From my experience, it appears that providing Corinna Vinschen (or better put: the Cygwin developers in general) with a sound patch gets things fixed pretty timely.
>>
>> And since `core.createObject = rename` seems to work around the problem, it should be possible to patch the Cygwin runtime accordingly. Sure, it will take a little investigation *what* code should be changed, and how, but the obvious benefit to *all* Cygwin applications should make that effort worth your while.
> 
> Hmm. I'm not sure what a Cygwin fix would look like here. As I
> understand it, using link(2) will fail if the target exists, while
> using rename(2) will just clobber a target file.

Thanks to you and Junio for explaining why my idea was bad. Sorry!

Ciao,
Johannes