fast-import fails in read-only tree

fast-import fails in read-only tree

[Stefan Monnier]

I recently discovered that "git fast-import" signals an error if used in
a tree to which we do not have write-access, because it tries to create
a "objects/pack/tmp_pack_XXX" file even before starting to process
the commands.

Usually this is not a problem (we'll create new commits and such, so
write-access is indeed necessary), but in my case I was using
fast-import only for its "reading" operations (in order to combine
several inter-dependent "cat-file" operations into a single git
session).


        Stefan

Re: fast-import fails in read-only tree

[Jeff King]

On Thu, Jan 28, 2016 at 05:17:36PM -0500, Stefan Monnier wrote:

> I recently discovered that "git fast-import" signals an error if used in
> a tree to which we do not have write-access, because it tries to create
> a "objects/pack/tmp_pack_XXX" file even before starting to process
> the commands.
> 
> Usually this is not a problem (we'll create new commits and such, so
> write-access is indeed necessary), but in my case I was using
> fast-import only for its "reading" operations (in order to combine
> several inter-dependent "cat-file" operations into a single git
> session).

The primary goal of fast-import is to write that packfile. It kind of
sounds like you are using the wrong tool for the job.

Can you elaborate on what you are sending to fast-import (preferably
with a concrete example)? There may be a way to accomplish the same
thing with read-only tools like cat-file.

-Peff

Re: fast-import fails in read-only tree

[Stefan Monnier]

>> I recently discovered that "git fast-import" signals an error if used in
>> a tree to which we do not have write-access, because it tries to create
>> a "objects/pack/tmp_pack_XXX" file even before starting to process
>> the commands.
> The primary goal of fast-import is to write that packfile.  It kind of
> sounds like you are using the wrong tool for the job.

Yes, I realize that.  But in some cases it's the best tool available.
`fast-import' is very close to being a "generic access API" which can be
used instead of something like libgit.  I think it'd be good to push it
yet a bit closer.

My earlier "cat-blob applied to a tree" issue is another such case.

> Can you elaborate on what you are sending to fast-import (preferably
> with a concrete example)?

I'm sending a stream of "progress <foo>; cat-blob <foo>", basically.

The concrete example is in [BuGit](https://gitlab.com/monnier/bugit),
see for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c

> There may be a way to accomplish the same thing with read-only tools
> like cat-file.

Yes, I switched to using "cat-file --batch" instead, but it's less
convenient (I can't intersperse ad-hoc info in the output, the way I can
with "progress" in fast-import) and there are cases where the list of
files I need to extract cannot be determined without first looking at
some of those extracted files (I currently have been able to avoid
this in BuGit, luckily).

If I could use "cat-blob" on directories, there would be even more cases
where I'd want to use fast-import for read-only operations to reduce the
number of Git processes I fork.


        Stefan

Re: fast-import fails in read-only tree

[Jeff King]

On Fri, Jan 29, 2016 at 09:28:44AM -0500, Stefan Monnier wrote:

> > The primary goal of fast-import is to write that packfile.  It kind of
> > sounds like you are using the wrong tool for the job.
> 
> Yes, I realize that.  But in some cases it's the best tool available.
> `fast-import' is very close to being a "generic access API" which can be
> used instead of something like libgit.  I think it'd be good to push it
> yet a bit closer.

I'm not sure I agree. Git tries to make its innards available via
flexible plumbing commands. If we're not succeeding, I think that should
be fixed, rather than trying to shoe-horn an unrelated command to do the
job, even if it would be less code.

> > Can you elaborate on what you are sending to fast-import (preferably
> > with a concrete example)?
> 
> I'm sending a stream of "progress <foo>; cat-blob <foo>", basically.
> 
> The concrete example is in [BuGit](https://gitlab.com/monnier/bugit),
> see for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c

You can use custom cat-file formatting to output your "name" strings as
part of the same field. IOW, something like:

  git cat-file -p refs/heads/bugit-master:numbers |
  awk '{print $3 " " $4 }' |
  git cat-file --batch="%(rest)" |
  while read number; do
    read id; # assuming blob contents are single-line
    read _junk; # assumes blob ended in its own newline
    $fun "$id" "$number"
  done

That's from a fairly cursory reading of that bugit patch, though, so I
might be missing some requirement.

> Yes, I switched to using "cat-file --batch" instead, but it's less
> convenient (I can't intersperse ad-hoc info in the output, the way I can
> with "progress" in fast-import) and there are cases where the list of
> files I need to extract cannot be determined without first looking at
> some of those extracted files (I currently have been able to avoid
> this in BuGit, luckily).

I think the example above should handle the "intersperse" thing.

If you're really going to do a lot of interactive back-and-forth access
of objects, though, I think you want to set up pipes to cat-file. It's a
little tedious to allocate fifos, but something like:

  mkfifo in out
  (exec git cat-file --batch <in >out) &
  exec 8>in
  exec 9<out
  echo $sha >&8
  read mode type size <&9
  read content ;# or read $size, or read until newline
  echo $content >&8 ;# imagine content is another sha to look up
  ...read from &9, etc..

The fifos and numbered descriptors are annoying, but that's shell for
you. I suspect using "fast-import" wouldn't be much different.

One feature I do think would be useful (and almost implemented when I
added --batch-check=<format>) is a formatter for the object content,
with a pretty modifier. I.e., it would be nice to do:

  echo $some_tree |
  git cat-file --batch-check="%(objectsize:pretty) %(contents:pretty)"

to work as the rough equivalent of "git cat-file -p" (but here you could
feed multiple trees and get multiple answers).

-Peff

Re: fast-import fails in read-only tree

[Andreas Schwab]

Jeff King <peff@peff.net> writes:

> If you're really going to do a lot of interactive back-and-forth access
> of objects, though, I think you want to set up pipes to cat-file. It's a
> little tedious to allocate fifos, but something like:

With bash's coproc it's a bit less tedious:

>   mkfifo in out
>   (exec git cat-file --batch <in >out) &
>   exec 8>in
>   exec 9<out
>   echo $sha >&8
>   read mode type size <&9

    coproc CAT_FILE git cat-file --batch
    echo $sha >&${CAT_FILE[1]}
    read mode type size <&${CAT_FILE[0]}

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: fast-import fails in read-only tree

[Stefan Monnier]

> You can use custom cat-file formatting to output your "name" strings as
> part of the same field. IOW, something like:
[...]
> If you're really going to do a lot of interactive back-and-forth access
> of objects, though, I think you want to set up pipes to cat-file.

OMG, I didn't realize that cat-file doesn't buffer its output so it can
be read&write to/from the same process.  And the "%(rest)" thingy takes
care of the rest of my needs, indeed.

Thanks!

> It's a little tedious to allocate fifos, but something like:

That's not a problem.

> One feature I do think would be useful (and almost implemented when I
> added --batch-check=<format>) is a formatter for the object content,
> with a pretty modifier. I.e., it would be nice to do:
>
>   echo $some_tree |
>   git cat-file --batch-check="%(objectsize:pretty) %(contents:pretty)"
>
> to work as the rough equivalent of "git cat-file -p" (but here you could
> feed multiple trees and get multiple answers).

Yes, that would be a good improvement,


        Stefan