Races on ref .lock files?

Races on ref .lock files?

[Andreas Krey]

Hi all,

We're occasionally seeing a lot of 

  error: cannot lock ref 'stash-refs/pull-requests/18978/to': Unable to create '/opt/apps/..../repositories/68/stash-refs/pull-requests/18978/to.lock': File exists.

from the server side with fetches as well as pushes. (Bitbucket server.)

What I find strange is that neither the fetches nor the pushes even
touch these refs (but the bitbucket triggers underneath might).

But my question is whether there are race conditions that can cause
such messages in regular operation - they continue with 'If no other git
process is currently running, this probably means a git process crashed
in this repository earlier.' which indicates some level of anticipation.

- Andreas

-- 
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800

Re: Races on ref .lock files?

[Junio C Hamano]

Andreas Krey <a.krey@gmx.de> writes:

> We're occasionally seeing a lot of 
>
>   error: cannot lock ref 'stash-refs/pull-requests/18978/to': Unable to create '/opt/apps/..../repositories/68/stash-refs/pull-requests/18978/to.lock': File exists.
>
> from the server side with fetches as well as pushes. (Bitbucket server.)
>
> What I find strange is that neither the fetches nor the pushes even
> touch these refs (but the bitbucket triggers underneath might).
>
> But my question is whether there are race conditions that can cause
> such messages in regular operation - they continue with 'If no other git
> process is currently running, this probably means a git process crashed
> in this repository earlier.' which indicates some level of anticipation.

I think (and I think you also think) these messages come from the
Bitbucket side, not your "git push" (or "git fetch").  Not having
seen Bitbucket's sources, I can only guess, but assuming that its
pull-request is triggered from their Web frontend like GitHub's
does, it is quite possible when you try to "push" into (or "fetch"
from, for that matter) a repository, somebody is clicking a button
to create that ref.  We do not know what their "receive-pack" that
responds to your "git push" (or "upload-pack" for "git fetch") does
when there are locked refs.  I'd naively think that unless you are
pushing to that ref you showed an error message for, the receiving
end shouldn't care if the ref is being written by somebody else, but
who knows ;-) They may have their own reasons wanting to lock that
ref that we think would be irrelevant for the operation, causing
errors.

Re: Races on ref .lock files?

[Bryan Turner]

Andreas,

Bitbucket Server developer here. Typically these errors on your client
are indicative of git gc --auto being triggered by git-receive-pack on
the server. Auto GC directly attached to a push in a repository with
pull requests often fails due to concurrent ref updates linked to
background pull request processing.

If you'd like to investigate more in depth, I'd encourage you to
create a ticket on support.atlassian.com so we can work with you.
Otherwise, if you just want to prevent seeing these messages, you can
either fork the relevant repository in Bitbucket Server (which
disables auto GC), or run "git config gc.auto 0" in
/opt/apps/.../repositories/68. Once auto GC is disabled, Bitbucket
Server will automatically take over managing GC for the repository
without any additional configuration required.

Note that we're working on revamping our GC handling such that auto GC
will always be disabled for all repositories and managed explicitly
within Bitbucket Server instead, so a future upgrade should
automatically prevent these messages from appearing on clients.

Best regards,
Bryan Turner

On Fri, Dec 16, 2016 at 9:20 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Andreas Krey <a.krey@gmx.de> writes:
>
>> We're occasionally seeing a lot of
>>
>>   error: cannot lock ref 'stash-refs/pull-requests/18978/to': Unable to create '/opt/apps/..../repositories/68/stash-refs/pull-requests/18978/to.lock': File exists.
>>
>> from the server side with fetches as well as pushes. (Bitbucket server.)
>>
>> What I find strange is that neither the fetches nor the pushes even
>> touch these refs (but the bitbucket triggers underneath might).
>>
>> But my question is whether there are race conditions that can cause
>> such messages in regular operation - they continue with 'If no other git
>> process is currently running, this probably means a git process crashed
>> in this repository earlier.' which indicates some level of anticipation.
>
> I think (and I think you also think) these messages come from the
> Bitbucket side, not your "git push" (or "git fetch").  Not having
> seen Bitbucket's sources, I can only guess, but assuming that its
> pull-request is triggered from their Web frontend like GitHub's
> does, it is quite possible when you try to "push" into (or "fetch"
> from, for that matter) a repository, somebody is clicking a button
> to create that ref.  We do not know what their "receive-pack" that
> responds to your "git push" (or "upload-pack" for "git fetch") does
> when there are locked refs.  I'd naively think that unless you are
> pushing to that ref you showed an error message for, the receiving
> end shouldn't care if the ref is being written by somebody else, but
> who knows ;-) They may have their own reasons wanting to lock that
> ref that we think would be irrelevant for the operation, causing
> errors.
>
>
>

Re: Races on ref .lock files?

[Andreas Krey]

On Fri, 16 Dec 2016 15:34:22 +0000, Bryan Turner wrote:
...
> Bitbucket Server developer here.

Social media rock. :-)

> If you'd like to investigate more in depth, I'd encourage you to
> create a ticket on support.atlassian.com so we can work with you.

That is going to be postponed until we update our bitbucket instance
to the current state.

> Otherwise, if you just want to prevent seeing these messages, you can
> either fork the relevant repository in Bitbucket Server (which
> disables auto GC), or run "git config gc.auto 0" in

Doing that for now. Will come back either if it doesn't help,
or after the upgrade.

> within Bitbucket Server instead, so a future upgrade should
> automatically prevent these messages from appearing on clients.

I still wonder if git itself should prevent these, or is there
a (git level) recommendation not to enable auto-gc in repos where
people regularly push to?

- Andreas

-- 
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800

Re: Races on ref .lock files?

[Andreas Krey]

On Fri, 16 Dec 2016 09:20:07 +0000, Junio C Hamano wrote:
...
> >   error: cannot lock ref 'stash-refs/pull-requests/18978/to': Unable to create '/opt/apps/..../repositories/68/stash-refs/pull-requests/18978/to.lock': File exists.
...
> I think (and I think you also think) these messages come from the
> Bitbucket side, not your "git push" (or "git fetch").

I *know* that this is the case - we don't have refs like that
on the local side. (Our users are more scared about them.)

...
> when there are locked refs.  I'd naively think that unless you are
> pushing to that ref you showed an error message for, the receiving
> end shouldn't care if the ref is being written by somebody else, but
> who knows ;-) They may have their own reasons wanting to lock that
> ref that we think would be irrelevant for the operation, causing
> errors.

Possible. I'm going Byrans way for now, disabling the gc there.

But:

In a different instance, we have a simple bare git repo that we
use for backup purposes. Which means there are lots of pushes
going there (all to disjunct refs), and I now cared to look
into those logfiles:

----snip
Wed Dec 21 05:08:14 CET 2016
fatal: Unable to create '/data/git-backup/backup.git/packed-refs.lock': File exists.

If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue.
error: failed to run pack-refs
To git-backup-user@socrepo.advantest.com:backup.git
 + 8aac9ae...2df6d56 refs/zz/current -> refs/backup/socvm217/ZworkspacesZsocvm217ZjohanabtZws-release_tools.Ycurr (forced update)
----snip

I interpret this as "I updated the refs files, but packing them
didn't work because someone else was also packing right now."

Is that happening as designed, or do I need to be afraid
that some refs didn't make the push?

To ask differently, is git relying on people reading such
messages and following up on them? And thus isn't that
easy to use in automated processes? (Additional problem:
The user in question, besides being an automat, doesn't
have the capability to work in the target repository.)

Andreas

-- 
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800

Re: Races on ref .lock files?

[Junio C Hamano]

Andreas Krey <a.krey@gmx.de> writes:

> In a different instance, we have a simple bare git repo that we
> use for backup purposes. Which means there are lots of pushes
> going there (all to disjunct refs), and I now cared to look
> into those logfiles:
>
> ----snip
> Wed Dec 21 05:08:14 CET 2016
> fatal: Unable to create '/data/git-backup/backup.git/packed-refs.lock': File exists.
>
> If no other git process is currently running, this probably means a
> git process crashed in this repository earlier. Make sure no other git
> process is running and remove the file manually to continue.
> error: failed to run pack-refs
> To git-backup-user@socrepo.advantest.com:backup.git
>  + 8aac9ae...2df6d56 refs/zz/current -> refs/backup/socvm217/ZworkspacesZsocvm217ZjohanabtZws-release_tools.Ycurr (forced update)
> ----snip
>
> I interpret this as "I updated the refs files, but packing them
> didn't work because someone else was also packing right now."

Correct.

> Is that happening as designed, or do I need to be afraid
> that some refs didn't make the push?

Correct and No.  Packing refs into the packed-refs file is merely a
performance thing and done under the lock (needless to say, updating
individual refs is also done under the lock).  Your push may have
competed with somebody else's push that started earlier and you may
have given up packing refs, but no ill effect should be left behind.

When the lock holder (the other guy who competes with your push)
stuffs refs into a packed-refs file, the values for the refs you
pushed may not be in the packed-refs file, because the other guy may
have observed and captured the value before your push updated them.
Those refs updated by you that are missed by the other guy will be
left as loose refs.  Because whenever Git tries to find the value
for a ref, it always checks the loose refs first, there is no issue
due to this.