not all tests pass when on an NFS partition

not all tests pass when on an NFS partition

[E R]

When I run 'make test' for git-1.6.1.3 it will fail on this test when
running on an NFS partition.
Is this a known problem? Is this a real problem?

My uname -a: Linux XXX 2.6.18-92.1.17.el5 #1 SMP Tue Nov 4 13:45:01
EST 2008 i686 athlon i386 GNU/Linux


*** t5304-prune.sh ***
*   ok 1: setup
*   ok 2: prune stale packs
*   ok 3: prune --expire
* FAIL 4: gc: implicit prune --expire


                before=$(git count-objects | sed "s/ .*//") &&
                BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
                BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
                test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
                test -f $BLOB_FILE &&
                test-chmtime =-$((86400*14-30)) $BLOB_FILE &&
                git gc &&
                test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
                test -f $BLOB_FILE &&
                test-chmtime =-$((86400*14+1)) $BLOB_FILE &&
                git gc &&
                test $before = $(git count-objects | sed "s/ .*//") &&
                ! test -f $BLOB_FILE

Re: not all tests pass when on an NFS partition

[Junio C Hamano]

E R <pc88mxer@gmail.com> writes:

> When I run 'make test' for git-1.6.1.3 it will fail on this test when
> running on an NFS partition.

... when the filesystem clock and the machine clock are out of sync.

> Is this a known problem? Is this a real problem?

It's a problem whose cause can be easily guessed ;-)

The tests check if the default expiration period (14 days) are honored by
creating unreferenced objects that are slightly older and slightly newer
than the cut-off timestamp and running the gc.  I suspect that the issue
with network filesystems whose clock are not grossly skewed can easily be
worked around by tweaking these -30 and +1 constants to larger (but not
too large to make the tests meaningless) values.

> My uname -a: Linux XXX 2.6.18-92.1.17.el5 #1 SMP Tue Nov 4 13:45:01
> EST 2008 i686 athlon i386 GNU/Linux
>
>
> *** t5304-prune.sh ***
> *   ok 1: setup
> *   ok 2: prune stale packs
> *   ok 3: prune --expire
> * FAIL 4: gc: implicit prune --expire
>
>
>                 before=$(git count-objects | sed "s/ .*//") &&
>                 BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
>                 BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
>                 test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
>                 test -f $BLOB_FILE &&
>                 test-chmtime =-$((86400*14-30)) $BLOB_FILE &&
>                 git gc &&
>                 test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
>                 test -f $BLOB_FILE &&
>                 test-chmtime =-$((86400*14+1)) $BLOB_FILE &&
>                 git gc &&
>                 test $before = $(git count-objects | sed "s/ .*//") &&
>                 ! test -f $BLOB_FILE

Re: not all tests pass when on an NFS partition

[Brandon Casey]

Junio C Hamano wrote:
> E R <pc88mxer@gmail.com> writes:
> 
>> When I run 'make test' for git-1.6.1.3 it will fail on this test when
>> running on an NFS partition.
> 
> ... when the filesystem clock and the machine clock are out of sync.

I previously modified t5304 to make test-chmtime modify the mtime of
the file based on the system time rather than the file mtime precisely to
avoid this problem.  i.e. the argument to test-chmtime is now prefixed with '='

So I don't think it is related to out of sync clock on NFS server and client.
There may be a bug in the NFS client though.

>> Is this a known problem? Is this a real problem?

Known by me, but I haven't mentioned it since I haven't had a moment to
diagnose it.

>> My uname -a: Linux XXX 2.6.18-92.1.17.el5 #1 SMP Tue Nov 4 13:45:01
>> EST 2008 i686 athlon i386 GNU/Linux

RHEL 5.2? me too.

RHEL 4.7 works fine, same NFS server (running 5.2).

-brandon

Re: not all tests pass when on an NFS partition

[E R]

On Thu, Feb 12, 2009 at 6:54 PM, Brandon Casey <casey@nrlssc.navy.mil> wrote:

> RHEL 5.2? me too.
>
> RHEL 4.7 works fine, same NFS server (running 5.2).

Yes - it's Centos 5.2. Very interesting.

Re: not all tests pass when on an NFS partition

[E R]

Btw - what kind of risks are involved in using git when test
t5304-prune.sh doesn't pass? Do we risk repository corruption?

On Sun, Feb 15, 2009 at 2:19 PM, E R <pc88mxer@gmail.com> wrote:
> On Thu, Feb 12, 2009 at 6:54 PM, Brandon Casey <casey@nrlssc.navy.mil> wrote:
>
>> RHEL 5.2? me too.
>>
>> RHEL 4.7 works fine, same NFS server (running 5.2).
>
> Yes - it's Centos 5.2. Very interesting.
>

Re: not all tests pass when on an NFS partition

[Brandon Casey]

E R wrote:
> On Thu, Feb 12, 2009 at 6:54 PM, Brandon Casey <casey@nrlssc.navy.mil> wrote:
> 
>> RHEL 5.2? me too.
>>
>> RHEL 4.7 works fine, same NFS server (running 5.2).
> 
> Yes - it's Centos 5.2. Very interesting.


It looks like mtime (and ctime) are being filled in with the atime value.

I modified builtin-prune.c as follows (warning: copy/pasted):

diff --git a/builtin-prune.c b/builtin-prune.c
index 545e9c1..2c1d351 100644
--- a/builtin-prune.c
+++ b/builtin-prune.c
@@ -38,8 +38,17 @@ static int prune_object(char *path, const char *filename, con
                struct stat st;
                if (lstat(fullpath, &st))
                        return error("Could not stat '%s'", fullpath);
-               if (st.st_mtime > expire)
+               printf("%s: expire %lu\n"
+                      "    inode  %lu\n"
+                      "    atime  %lu\n"
+                      "    mtime  %lu\n"
+                      "    ctime  %lu\n",
+                      fullpath, expire,
+                      st.st_ino, st.st_atime, st.st_mtime, st.st_ctime);
+               if (st.st_mtime > expire) {
+                       puts("not expiring");
                        return 0;
+                }
        }
        if (show_only || verbose) {
                enum object_type type = sha1_object_info(sha1, NULL);


After inserting an 'exit' statement before test 4 of t5304, the second
'git gc' prints the following:

    $ git gc
    Counting objects: 3, done.
    Writing objects: 100% (3/3), done.
    Total 3 (delta 0), reused 3 (delta 0)
    .git/objects/fb/f6228a25d50b1ea329e06e75b2f3f1de3793a0: expire 1233599127
        inode  46186578
        atime  1234807371
        mtime  1234807371
        ctime  1234807371
    not expiring

And stat gives:

    $ stat .git/objects/fb/f6228a25d50b1ea329e06e75b2f3f1de3793a0
      File: `.git/objects/fb/f6228a25d50b1ea329e06e75b2f3f1de3793a0'
      Size: 23              Blocks: 16         IO Block: 4096   regular file
    Device: 1ah/26d Inode: 46186578    Links: 1
    Access: (0444/-r--r--r--)  Uid: (  XXX)   Gid: (  XXX)
    Access: 2009-02-16 12:02:51.000000000 -0600
    Modify: 2009-02-02 12:03:45.000000000 -0600
    Change: 2009-02-16 12:03:46.000000000 -0600

And for the lazy

    $ perl -e 'print scalar(localtime(1234807371)), "\n";'
    Mon Feb 16 12:02:51 2009


Compiling with -O0 produces the same results.
Also, I am unable to write a program to reproduce this behavior.

Here's a variation for you. 'git prune --expire 2.weeks.ago' only fails if
called after a failed 'git gc'.

# edit t5304-prune.sh to replace the second 'git gc' in test 4 with
# 'git prune --expire 2.weeks.ago'
$ rm -rf 'trash directory.t5304-prune'
$ ./t5304-prune.sh
# Success for test 4, but the new 11 and 12 still fail.
# Plus, the correct values for atime, mtime, and ctime are printed out by
# the modifications in the patch above.


# Reset t5304-prune.sh back to the original, then add an exit statement
# _after_ test 4.  The test will fail.  Then cd into 'trash directory.t5304-prune'
# and again 'git prune --expire 2.weeks.ago' will succeed.
$ rm -rf 'trash directory.t5304-prune'
$ ./t5304-prune.sh
$ cd 'trash directory.t5304-prune'
$ git prune --expire 2.weeks.ago
# success, correct file times are printed out.

# Now do the same thing as above, but this time run 'git gc' before running
# prune.
$ rm -rf 'trash directory.t5304-prune'
$ ./t5304-prune.sh
$ cd 'trash directory.t5304-prune'
$ git gc
# fails to remove appropriate file, all file times are from atime
$ git prune --expire 2.weeks.ago
# fails to remove the appropriate file, all file times are from atime

After some amount of time (minutes), git gc succeeds in removing the
object and prints out the correct time values.

-brandon

Re: not all tests pass when on an NFS partition

[Brandon Casey]

E R wrote:
> Btw - what kind of risks are involved in using git when test
> t5304-prune.sh doesn't pass? Do we risk repository corruption?

For the case in the test, git-prune is failing to remove an object
that it should, which won't cause repo corruption.  It will just
cause retention of an object for longer than it is needed. I suspect
this is caused by some flaw in the NFS client (disclosure: IANAKH).

In practice, I don't think you'll experience this flaw.  I think it
shows up in the test because we are manually adjusting the mtime of
the objects and then immediately trying to prune them and this must
tweak whatever code has a bug.

I have been using git on 5.2 for a couple of months now with no problem.
Admittedly, prune is a rare operation, usually triggered by git-gc.

-brandon