* fast-import fails in read-only tree @ 2016-01-28 22:17 Stefan Monnier 2016-01-29 6:08 ` Jeff King 0 siblings, 1 reply; 6+ messages in thread From: Stefan Monnier @ 2016-01-28 22:17 UTC (permalink / raw) To: git I recently discovered that "git fast-import" signals an error if used in a tree to which we do not have write-access, because it tries to create a "objects/pack/tmp_pack_XXX" file even before starting to process the commands. Usually this is not a problem (we'll create new commits and such, so write-access is indeed necessary), but in my case I was using fast-import only for its "reading" operations (in order to combine several inter-dependent "cat-file" operations into a single git session). Stefan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: fast-import fails in read-only tree 2016-01-28 22:17 fast-import fails in read-only tree Stefan Monnier @ 2016-01-29 6:08 ` Jeff King 2016-01-29 14:28 ` Stefan Monnier 0 siblings, 1 reply; 6+ messages in thread From: Jeff King @ 2016-01-29 6:08 UTC (permalink / raw) To: Stefan Monnier; +Cc: git On Thu, Jan 28, 2016 at 05:17:36PM -0500, Stefan Monnier wrote: > I recently discovered that "git fast-import" signals an error if used in > a tree to which we do not have write-access, because it tries to create > a "objects/pack/tmp_pack_XXX" file even before starting to process > the commands. > > Usually this is not a problem (we'll create new commits and such, so > write-access is indeed necessary), but in my case I was using > fast-import only for its "reading" operations (in order to combine > several inter-dependent "cat-file" operations into a single git > session). The primary goal of fast-import is to write that packfile. It kind of sounds like you are using the wrong tool for the job. Can you elaborate on what you are sending to fast-import (preferably with a concrete example)? There may be a way to accomplish the same thing with read-only tools like cat-file. -Peff ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: fast-import fails in read-only tree 2016-01-29 6:08 ` Jeff King @ 2016-01-29 14:28 ` Stefan Monnier 2016-01-30 5:13 ` Jeff King 0 siblings, 1 reply; 6+ messages in thread From: Stefan Monnier @ 2016-01-29 14:28 UTC (permalink / raw) To: git >> I recently discovered that "git fast-import" signals an error if used in >> a tree to which we do not have write-access, because it tries to create >> a "objects/pack/tmp_pack_XXX" file even before starting to process >> the commands. > The primary goal of fast-import is to write that packfile. It kind of > sounds like you are using the wrong tool for the job. Yes, I realize that. But in some cases it's the best tool available. `fast-import' is very close to being a "generic access API" which can be used instead of something like libgit. I think it'd be good to push it yet a bit closer. My earlier "cat-blob applied to a tree" issue is another such case. > Can you elaborate on what you are sending to fast-import (preferably > with a concrete example)? I'm sending a stream of "progress <foo>; cat-blob <foo>", basically. The concrete example is in [BuGit](https://gitlab.com/monnier/bugit), see for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c > There may be a way to accomplish the same thing with read-only tools > like cat-file. Yes, I switched to using "cat-file --batch" instead, but it's less convenient (I can't intersperse ad-hoc info in the output, the way I can with "progress" in fast-import) and there are cases where the list of files I need to extract cannot be determined without first looking at some of those extracted files (I currently have been able to avoid this in BuGit, luckily). If I could use "cat-blob" on directories, there would be even more cases where I'd want to use fast-import for read-only operations to reduce the number of Git processes I fork. Stefan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: fast-import fails in read-only tree 2016-01-29 14:28 ` Stefan Monnier @ 2016-01-30 5:13 ` Jeff King 2016-01-30 9:05 ` Andreas Schwab 2016-01-30 13:56 ` Stefan Monnier 0 siblings, 2 replies; 6+ messages in thread From: Jeff King @ 2016-01-30 5:13 UTC (permalink / raw) To: Stefan Monnier; +Cc: git On Fri, Jan 29, 2016 at 09:28:44AM -0500, Stefan Monnier wrote: > > The primary goal of fast-import is to write that packfile. It kind of > > sounds like you are using the wrong tool for the job. > > Yes, I realize that. But in some cases it's the best tool available. > `fast-import' is very close to being a "generic access API" which can be > used instead of something like libgit. I think it'd be good to push it > yet a bit closer. I'm not sure I agree. Git tries to make its innards available via flexible plumbing commands. If we're not succeeding, I think that should be fixed, rather than trying to shoe-horn an unrelated command to do the job, even if it would be less code. > > Can you elaborate on what you are sending to fast-import (preferably > > with a concrete example)? > > I'm sending a stream of "progress <foo>; cat-blob <foo>", basically. > > The concrete example is in [BuGit](https://gitlab.com/monnier/bugit), > see for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c You can use custom cat-file formatting to output your "name" strings as part of the same field. IOW, something like: git cat-file -p refs/heads/bugit-master:numbers | awk '{print $3 " " $4 }' | git cat-file --batch="%(rest)" | while read number; do read id; # assuming blob contents are single-line read _junk; # assumes blob ended in its own newline $fun "$id" "$number" done That's from a fairly cursory reading of that bugit patch, though, so I might be missing some requirement. > Yes, I switched to using "cat-file --batch" instead, but it's less > convenient (I can't intersperse ad-hoc info in the output, the way I can > with "progress" in fast-import) and there are cases where the list of > files I need to extract cannot be determined without first looking at > some of those extracted files (I currently have been able to avoid > this in BuGit, luckily). I think the example above should handle the "intersperse" thing. If you're really going to do a lot of interactive back-and-forth access of objects, though, I think you want to set up pipes to cat-file. It's a little tedious to allocate fifos, but something like: mkfifo in out (exec git cat-file --batch <in >out) & exec 8>in exec 9<out echo $sha >&8 read mode type size <&9 read content ;# or read $size, or read until newline echo $content >&8 ;# imagine content is another sha to look up ...read from &9, etc.. The fifos and numbered descriptors are annoying, but that's shell for you. I suspect using "fast-import" wouldn't be much different. One feature I do think would be useful (and almost implemented when I added --batch-check=<format>) is a formatter for the object content, with a pretty modifier. I.e., it would be nice to do: echo $some_tree | git cat-file --batch-check="%(objectsize:pretty) %(contents:pretty)" to work as the rough equivalent of "git cat-file -p" (but here you could feed multiple trees and get multiple answers). -Peff ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: fast-import fails in read-only tree 2016-01-30 5:13 ` Jeff King @ 2016-01-30 9:05 ` Andreas Schwab 2016-01-30 13:56 ` Stefan Monnier 1 sibling, 0 replies; 6+ messages in thread From: Andreas Schwab @ 2016-01-30 9:05 UTC (permalink / raw) To: Jeff King; +Cc: Stefan Monnier, git Jeff King <peff@peff.net> writes: > If you're really going to do a lot of interactive back-and-forth access > of objects, though, I think you want to set up pipes to cat-file. It's a > little tedious to allocate fifos, but something like: With bash's coproc it's a bit less tedious: > mkfifo in out > (exec git cat-file --batch <in >out) & > exec 8>in > exec 9<out > echo $sha >&8 > read mode type size <&9 coproc CAT_FILE git cat-file --batch echo $sha >&${CAT_FILE[1]} read mode type size <&${CAT_FILE[0]} Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: fast-import fails in read-only tree 2016-01-30 5:13 ` Jeff King 2016-01-30 9:05 ` Andreas Schwab @ 2016-01-30 13:56 ` Stefan Monnier 1 sibling, 0 replies; 6+ messages in thread From: Stefan Monnier @ 2016-01-30 13:56 UTC (permalink / raw) To: git > You can use custom cat-file formatting to output your "name" strings as > part of the same field. IOW, something like: [...] > If you're really going to do a lot of interactive back-and-forth access > of objects, though, I think you want to set up pipes to cat-file. OMG, I didn't realize that cat-file doesn't buffer its output so it can be read&write to/from the same process. And the "%(rest)" thingy takes care of the rest of my needs, indeed. Thanks! > It's a little tedious to allocate fifos, but something like: That's not a problem. > One feature I do think would be useful (and almost implemented when I > added --batch-check=<format>) is a formatter for the object content, > with a pretty modifier. I.e., it would be nice to do: > > echo $some_tree | > git cat-file --batch-check="%(objectsize:pretty) %(contents:pretty)" > > to work as the rough equivalent of "git cat-file -p" (but here you could > feed multiple trees and get multiple answers). Yes, that would be a good improvement, Stefan ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-01-30 13:57 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-01-28 22:17 fast-import fails in read-only tree Stefan Monnier 2016-01-29 6:08 ` Jeff King 2016-01-29 14:28 ` Stefan Monnier 2016-01-30 5:13 ` Jeff King 2016-01-30 9:05 ` Andreas Schwab 2016-01-30 13:56 ` Stefan Monnier
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.