Re: [PATCH 0/2] Sparse index: fetch, pull, ls-files

From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Cc: git@vger.kernel.org, gitster@pobox.com, newren@gmail.com,
	vdye@github.com, Derrick Stolee <derrickstolee@github.com>
Subject: Re: [PATCH 0/2] Sparse index: fetch, pull, ls-files
Date: Tue, 23 Nov 2021 02:57:33 +0100	[thread overview]
Message-ID: <211123.86lf1fwrq5.gmgdl@evledraar.gmail.com> (raw)
In-Reply-To: <pull.1080.git.1637077083.gitgitgadget@gmail.com>

On Tue, Nov 16 2021, Derrick Stolee via GitGitGadget wrote:

> The 'ls-files' builtin is useful when debugging the index, and some scripts
> use it, too. We are not changing the default behavior which expands a sparse
> index in order to show all of the cached blobs. Instead, we add a '--sparse'
> option that allows us to see the sparse directory entries upon request.
> Combined with --debug, we can see a lot of index details, such as:
>
> $ git ls-files --debug --sparse
> LICENSE
>   ctime: 1634910503:287405820
>   mtime: 1634910503:287405820
>   dev: 16777220 ino: 119325319
>   uid: 501  gid: 20
>   size: 1098    flags: 200000
> README.md
>   ctime: 1634910503:288090279
>   mtime: 1634910503:288090279
>   dev: 16777220 ino: 119325320
>   uid: 501  gid: 20
>   size: 934 flags: 200000
> bin/index.js
>   ctime: 1634910767:828434033
>   mtime: 1634910767:828434033
>   dev: 16777220 ino: 119325520
>   uid: 501  gid: 20
>   size: 7292    flags: 200000
> examples/
>   ctime: 0:0
>   mtime: 0:0
>   dev: 0    ino: 0
>   uid: 0    gid: 0
>   size: 0   flags: 40004000
> package.json
>   ctime: 1634910503:288676330
>   mtime: 1634910503:288676330
>   dev: 16777220 ino: 119325321
>   uid: 501  gid: 20
>   size: 680 flags: 200000
>
>
> (In this example, the 'examples/' directory is sparse.)
>
> Thanks!

That's useful, and seems to be closing the same feature gap that the RFC
series I sent back in March[1], but at the time you went with adding
this ability to t/helper/test-read-cache.c.

With your series diffing the two with the data used in your new test
shows that we had this with the helper beore:

     diff -u <(t/helper/test-tool -C sparse-index read-cache --table) <(t/helper/test-tool -C sparse-index read-cache --table --expand)
    --- /dev/fd/63  2021-11-23 02:57:00.980651400 +0100
    +++ /dev/fd/62  2021-11-23 02:57:00.980651400 +0100
    @@ -13,9 +13,13 @@
     100644 blob cebd1739abee3e524d72ca9f51465a94d5a71daf   e
     100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391   folder1-
     100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391   folder1.x
    -040000 tree e0f2d30b633eb781d675fedd78808135103fe1a0   folder1/
    +100644 blob 8b137891791fe96927ad78e64b0aad7bded08bdc   folder1/0/0/0
    +100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391   folder1/0/1
    +100644 blob 78981922613b2afb6025042ff6bd878ac1994e85   folder1/a
     100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391   folder10
    -040000 tree 123706f6fc38949628eaf0483edbf97ba21123ae   folder2/
    +100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391   folder2/0/0/0
    +100644 blob 8b137891791fe96927ad78e64b0aad7bded08bdc   folder2/0/1
    +100644 blob 78981922613b2afb6025042ff6bd878ac1994e85   folder2/a
     100644 blob 18912c9a915632d7b3344ec25f349dc8b4b9bf27   g
    -040000 tree aaff74984cccd156a469afa7d9ab10e4777beb24   x/
    +100644 blob 78981922613b2afb6025042ff6bd878ac1994e85   x/a
     100644 blob 2d8c856aebeb20da61bd5112d6fa46ff3f56a9e8   z

Which we can now get out of ls-files:

    $ diff -u <(git -C sparse-index ls-files --stage --sparse) <(git -C sparse-index ls-files --stage)
    --- /dev/fd/63  2021-11-23 02:55:13.329613255 +0100
    +++ /dev/fd/62  2021-11-23 02:55:13.329613255 +0100
    @@ -13,9 +13,13 @@
     100644 cebd1739abee3e524d72ca9f51465a94d5a71daf 0      e
     100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0      folder1-
     100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0      folder1.x
    -040000 e0f2d30b633eb781d675fedd78808135103fe1a0 0      folder1/
    +100644 8b137891791fe96927ad78e64b0aad7bded08bdc 0      folder1/0/0/0
    +100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0      folder1/0/1
    +100644 78981922613b2afb6025042ff6bd878ac1994e85 0      folder1/a
     100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0      folder10
    -040000 123706f6fc38949628eaf0483edbf97ba21123ae 0      folder2/
    +100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0      folder2/0/0/0
    +100644 8b137891791fe96927ad78e64b0aad7bded08bdc 0      folder2/0/1
    +100644 78981922613b2afb6025042ff6bd878ac1994e85 0      folder2/a
     100644 18912c9a915632d7b3344ec25f349dc8b4b9bf27 0      g
    -040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0      x/
    +100644 78981922613b2afb6025042ff6bd878ac1994e85 0      x/a
     100644 2d8c856aebeb20da61bd5112d6fa46ff3f56a9e8 0      z

So that's neat, but unless I'm wrong about there still being some
special-sauce in t/helper/test-read-cache.c that's needed this series
seems incomplete without migrating the existing test users of it to this
new ls-files --sparse, followed by a cherry-pick (or equivalent) of [2].

Is there any reason we wouldn't use ls-files instead of the test-tool in
the tests now? I understood from you at the time that the reason for the
that mode of the test-tool existing was an interim state until ls-files
learned to emit this sort of output, but that you wanted to add that
later. It seems we've arrived at that "later" :)

1. https://lore.kernel.org/git/20210317132814.30175-1-avarab@gmail.com/
2. https://lore.kernel.org/git/20210317132814.30175-5-avarab@gmail.com/