[PATCH] Quick description of possible gitattributes system

[PATCH] Quick description of possible gitattributes system

[Andy Parkins]

Junio was working on a gitattributes system.  Rather than be useful and
write code, I wrote this document to show how an attribute system might
work.

The ideas in this document come primarily from the "unresolved issues"
thread from the git mailing list.  gmane/28388.  In particular, from:
 - Brian Gernhardt
 - Johannes Schindelin
 - Junio Hamano
 - Linus Torvalds
 - Martin Waitz
 - Nicolas Pitre
 - Robin Rosenberg
 - Theodore Tso
So they deserve the credit for the ideas, and I deserve the blame for
the description.

Signed-off-by: Andy Parkins <andyparkins@gmail.com>
---
This one is better.  I've included some of the points raised in the
"unresolved issues" thread from January.

 Documentation/attributes.txt |  222 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 222 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/attributes.txt

diff --git a/Documentation/attributes.txt b/Documentation/attributes.txt
new file mode 100644
index 0000000..9877786
--- /dev/null
+++ b/Documentation/attributes.txt
@@ -0,0 +1,222 @@
+Git Attributes System
+=====================
+
+This document is a work-in-progress.  It's only purpose is to assemble
+attribute ideas into one place.
+
+== Attributes
+
+Individual paths are assigned attributes.  Typically they will only have
+one attribute, typically named after the corresponding mime type of that
+file.  However, it is not required that the attribute be a mime type nor
+is it required that this be the only attribute.
+
+Attributes are attached to paths in one of two ways:
+
+ - Via the .gitattributes file.  This will work in the same way as the
+   .gitignore file.  That is, there can be multiple .gitattributes files
+   throughout the working directory.
+
+   Typically the .gitattributes file will be a version controlled file,
+   just like .gitignore.
+
+   The format of this file will be similar to .gitignore.  "#" starting
+   a comment.  Globbing allowed.  Paths to subdirectories allowed.  Each
+   path will be followed by the list of attributes that applies to that
+   path.  Additionally, we would probably want a way of disabling an
+   attribute.  For example
+
+   -------------------------------------
+   # Sample .gitattributes file
+   *.png image/png
+   *.txt text/plain
+   *.c text/plain source-file
+   script/*.sh text/plain source-file
+   # Disable text/plain for a particular file
+   file-isnt-text-despite-extension.txt !text/plain
+   -------------------------------------
+
+ - Via the git configuration file system.  i.e. /etc/gitconfig,
+   $HOME/.gitconfig and $GIT_DIR/config.
+
+   This supplies the same facility as the .gitattributes file but allows
+   the attributes to be specifed out-of-tree.  This will give the
+   advantage that standard attributes can be specified per-system via
+   /etc/gitconfig; per-user via $HOME/config or per-repository via
+   $GIT_DIR/config.
+
+   It will obviously be of different format from the .gitattributes
+   file, but the attribute names and path specification will be
+   identical (and specified as if from the root of the repository tree).
+   For example:
+
+   ------------------------------------
+   [attributes "*.png"]
+       attribute = image/png
+   [attributes "*.c"]
+       attribute = text/plain
+	   attribute = source-file
+   ------------------------------------
+
+   (This is pretty ugly, because of the "*" and "." in the section name,
+   however I can't think of another way of doing this without resorting to
+   just copying the format of the .gitattributes file; like
+   -------------------------------------
+   [attributes]
+       path = *.png image/png
+	   path = *.txt text/plain
+	   path = *.c text/plain source-file
+   -------------------------------------
+   but perhaps people would like that more?)
+
+With these two in place, attributes for files can be specified and
+overridden easily.  The priority of the various files will be (from
+lowest to highest)
+
+ - /etc/gitconfig
+ - $HOME/.gitconfig
+ - .gitattributes
+ - $GIT_DIR/config
+
+The $GIT_DIR/config file is highest priority so that a user can always
+override an attribute that is in a .gitattributes managed by someone
+else.
+
+
+== Handlers
+
+At this point git would be in a position to answer the question "what
+attributes does this file have?" for any file in the tree.
+
+The handler system will allow us to specify operations that should be
+performed on files with particular attributes.  For example, we might
+want to have text files automatically filtered to match local line
+ending conditions.
+
+----------------------------------------
+[handler "fix-line-endings"]
+  attribute = text/plain
+  infilter = eol_lf
+  outfilter = eol_local
+----------------------------------------
+
+This definition creates a handler named "fix-line-endings" (don't suppose
+they actually need names) that would run the "eol_lf" filter on check
+in, and the "eol_local" filter on checkout.
+
+Similarly, we might want to tell git-show how to show images:
+
+----------------------------------------
+[handler "show-images"]
+  attribute = image/png
+  attribute = image/jpeg
+  attribute = image/gif
+  prettyfilter = pipe display -
+----------------------------------------
+
+With this filter in place,
+----------------------------------------
+ $ git-show v1.0.0:images/picture.png
+----------------------------------------
+Would run "display -" and pipe the picture to it.
+
+Multiple filters can be supplied, and order will be important.  So, to
+transparently decompress a file and fix the line endings for a text file
+we might do:
+
+---------------------------------------
+[handler "decompress-text"]
+  attribute = compressed-text
+  infilter = gunzip
+  infilter = eol_lf
+  outfilter = eol_local
+  outfilter = gzip
+---------------------------------------
+
+This might be useful for documentation files that come compressed (like
+README.gz used by Debian).
+
+Similarly, we might want to be able to look at compressed files stored
+in the repository:
+
+---------------------------------------
+[handler "show-compressed-file"]
+  attribute = application/gzip
+  prettyfilter = gunzip
+  prettyfilter = eol_auto
+---------------------------------------
+
+=== Handler Types
+
+ - infilter
+   Run by git-add when content is added to the repository.
+ - outfilter
+   Run by git-checkout or git-reset when copying content from the index
+   to the working directory.  (Actually, more likely this would be done
+   at low level in git-read-tree)
+ - prettyfilter
+   Run by git-cat-file or git-show when content is being pretty-printed
+   for display to the user.  If no prettyfilter is set, then it should
+   default to outfilter.
+ - merge
+   Git has built in, good support for merging standard text files.
+   However, if you wanted to add an external merge tool for merging,
+   say, XML files, you could supply it here
+ - diff
+   If you wanted to use an external diff for certain files, you would
+   specify it in in the diff handler
+
+
+
+=== Filter list
+
+ - eol_lf
+   Force all line endings to line-feed.
+
+ - eol_cr
+   Force all line endings to carriage-return.
+
+ - eol_crlf
+   Force all line endings to carriage-return, line-feed.
+
+ - eol_local
+   Force all line endings to local default (LF on UNIX; CRLF on Windows).
+
+ - eol_auto
+   Do the "right thing" depending on the file content and local default.
+
+ - pipe <cmd>
+   Open a bidirectional pipe and run <cmd>; which will transform the
+   file in whatever way one wishes.
+
+ - keywords
+   The traditional CVS keywords are substituted in the file.  Maybe:
+     $Id$, $Author$, $URL$, $Date$, $Rev$
+   This filter would behave differently depending on whether it was an
+   infilter or an outfilter.  On infilter it would remove the extra
+   information.  On outfilter it would add the extra information.
+
+ - gunzip, gzip / zip, unzip
+   Some files are stored compressed in their natural state (like ODF for
+   example).  This isn't optimum for getting the best out of git.  Git
+   (when making pack files) can store the delta between two files, and
+   compresses all objects in its store anyway.  So, a compressed text
+   file that changes over time will actually take up more disk space
+   than the uncompressed file would take up.  These compression filters
+   would fix that problem by transparently decompressing/compressing the
+   files on their way in and out.
+
+
+== Problems
+
+ - Initial Checkout
+   The .gitattributes file is not in the working directory, so how git
+   find out what to apply to a file?  The potential solution might be to
+   use the in-repo .gitattributes for checking out; and the working-tree
+   .gitattributes when checking in.
+ - Merge of .gitattributes
+   What happens when .gitattributes conflicts during a merge?
+ - Change of .gitattributes without change of files.
+   What happens when .gitattributes changes, but the files that would be
+   affected by that change do not?  Those files really should be checked
+   out again, to apply any new outfilter settings.
-- 
1.5.0.2.205.g74e20

Re: [PATCH] Quick description of possible gitattributes system

[Brian Gernhardt]

On Mar 1, 2007, at 7:06 AM, Andy Parkins wrote:

> The ideas in this document come primarily from the "unresolved issues"
> thread from the git mailing list.  gmane/28388.  In particular, from:
>  - Brian Gernhardt

I still get a kick out of seeing my name mentioned on the list.  :-)   
And if I'm getting some of the blame on me, (ew, get it off) I should  
probably put in my $0.02.

> +   For example:
> +
> +   ------------------------------------
> +   [attributes "*.png"]
> +       attribute = image/png
> +   [attributes "*.c"]
> +       attribute = text/plain
> +	   attribute = source-file
> +   ------------------------------------
> +
> +   (This is pretty ugly, because of the "*" and "." in the section  
> name,
> +   however I can't think of another way of doing this without  
> resorting to
> +   just copying the format of the .gitattributes file; like

Perhaps:

[attribute "image/png"]
    path = *.png
    show = "open %path%"
[attribute "text/plain"]
    path = *.c
    checkout = eol_to_local

I also think that this is a better way for handlers to be  
specified...  see below.

> +== Handlers
> +
> +At this point git would be in a position to answer the question "what
> +attributes does this file have?" for any file in the tree.
> +
> +The handler system will allow us to specify operations that should be
> +performed on files with particular attributes.  For example, we might
> +want to have text files automatically filtered to match local line
> +ending conditions.
> +
> +----------------------------------------
> +[handler "fix-line-endings"]
> +  attribute = text/plain
> +  infilter = eol_lf
> +  outfilter = eol_local
> +----------------------------------------
> +
> +This definition creates a handler named "fix-line-endings" (don't  
> suppose
> +they actually need names) that would run the "eol_lf" filter on check
> +in, and the "eol_local" filter on checkout.
> +

I think it would be better to directly associated these with the  
attribute, and have handler sections as shortcuts:

[handler "fix-line-endings"]
    infilter = eol_lf
    outfilter = eol_local
[attribute "text/plain"]
    handler = fix-line-endings
[attribute "text/source"]
    patch = *.source
    handler = fix-line-endings
    infilter = prettypatch

The file type (attribute, whatever) is the more user-important thing  
here.  It should, therefore, be the most visible thing when editing.   
I wouldn't want to have to read through the entirety of my config to  
find all of the handlers which affect a type, and it gathers all the  
options in one place.  Plus, it saves us from having to figure out in  
what order multiple handler sections on the same attribute get run  
in.  They get run in whatever order they're specified in the  
attribute's section.

> +=== Handler Types

[snip]

> +=== Filter list

[snip]

Both of these lists look right at first glance, although I think I  
prefer something simple like "show" to "prettyfilter".  And we might  
want a "reject" or "none" for merges...  Attempting to merge a bitmap  
is madness, so the system should just output a <file>- 
<head>.<extension> or similar.

> +== Problems
> +
> + - Initial Checkout
> +   The .gitattributes file is not in the working directory, so how  
> git
> +   find out what to apply to a file?  The potential solution might  
> be to
> +   use the in-repo .gitattributes for checking out; and the  
> working-tree
> +   .gitattributes when checking in.
> + - Merge of .gitattributes
> +   What happens when .gitattributes conflicts during a merge?

We may have to special case .gitattributes.  Force a checkout of it  
before any files in the directory or subdirectories, and not just on  
the initial checkout.  Checking out .gitattributes first during  
merges and pulls might be required and unmerged .gitattributes causes  
a big warning that it has to be resolved, and causes the rest of the  
repo to conform to the new attributes when it's resolved (possibly  
causing more merge conflicts).

> + - Change of .gitattributes without change of files.
> +   What happens when .gitattributes changes, but the files that  
> would be
> +   affected by that change do not?  Those files really should be  
> checked
> +   out again, to apply any new outfilter settings.

Actually, shouldn't the files also be run through the infilter to  
check for changes caused by that, too?

Maybe this is a more like a nickel,
~~ Brian

Re: [PATCH] Quick description of possible gitattributes system

[Robin Rosenberg]

torsdag 01 mars 2007 13:06 skrev Andy Parkins:
> Junio was working on a gitattributes system.  Rather than be useful and
> write code, I wrote this document to show how an attribute system might
> work.
> 
> The ideas in this document come primarily from the "unresolved issues"
> thread from the git mailing list.  gmane/28388.  In particular, from:
>  - Brian Gernhardt
>  - Johannes Schindelin
>  - Junio Hamano
>  - Linus Torvalds
>  - Martin Waitz
>  - Nicolas Pitre
>  - Robin Rosenberg
>  - Theodore Tso
> So they deserve the credit for the ideas, and I deserve the blame for
> the description.
> 
> Signed-off-by: Andy Parkins <andyparkins@gmail.com>
> ---
> This one is better.  I've included some of the points raised in the
> "unresolved issues" thread from January.
> 
>  Documentation/attributes.txt |  222 ++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 222 insertions(+), 0 deletions(-)
>  create mode 100644 Documentation/attributes.txt
> 
> diff --git a/Documentation/attributes.txt b/Documentation/attributes.txt
> new file mode 100644
> index 0000000..9877786
> --- /dev/null
> +++ b/Documentation/attributes.txt
> @@ -0,0 +1,222 @@
> +Git Attributes System
> +=====================
> +
> +This document is a work-in-progress.  It's only purpose is to assemble
> +attribute ideas into one place.
> +
> +== Attributes
> +
> +Individual paths are assigned attributes.  Typically they will only have
> +one attribute, typically named after the corresponding mime type of that
> +file.  However, it is not required that the attribute be a mime type nor
> +is it required that this be the only attribute.
> +
> +Attributes are attached to paths in one of two ways:
> +
> + - Via the .gitattributes file.  This will work in the same way as the
> +   .gitignore file.  That is, there can be multiple .gitattributes files
> +   throughout the working directory.
> +
> +   Typically the .gitattributes file will be a version controlled file,
> +   just like .gitignore.
> +
> +   The format of this file will be similar to .gitignore.  "#" starting
> +   a comment.  Globbing allowed.  Paths to subdirectories allowed.  Each
> +   path will be followed by the list of attributes that applies to that
> +   path.  Additionally, we would probably want a way of disabling an
> +   attribute.  For example
> +
> +   -------------------------------------
> +   # Sample .gitattributes file
> +   *.png image/png
> +   *.txt text/plain
> +   *.c text/plain source-file
> +   script/*.sh text/plain source-file
> +   # Disable text/plain for a particular file
> +   file-isnt-text-despite-extension.txt !text/plain
> +   -------------------------------------
> +
> + - Via the git configuration file system.  i.e. /etc/gitconfig,
> +   $HOME/.gitconfig and $GIT_DIR/config.
> +
> +   This supplies the same facility as the .gitattributes file but allows
> +   the attributes to be specifed out-of-tree.  This will give the
> +   advantage that standard attributes can be specified per-system via
> +   /etc/gitconfig; per-user via $HOME/config or per-repository via
> +   $GIT_DIR/config.
> +
> +   It will obviously be of different format from the .gitattributes
> +   file, but the attribute names and path specification will be
> +   identical (and specified as if from the root of the repository tree).
> +   For example:
> +
> +   ------------------------------------
> +   [attributes "*.png"]
> +       attribute = image/png
> +   [attributes "*.c"]
> +       attribute = text/plain
> +	   attribute = source-file
> +   ------------------------------------
> +
> +   (This is pretty ugly, because of the "*" and "." in the section name,
> +   however I can't think of another way of doing this without resorting to
> +   just copying the format of the .gitattributes file; like
> +   -------------------------------------
> +   [attributes]
> +       path = *.png image/png
> +	   path = *.txt text/plain
> +	   path = *.c text/plain source-file
> +   -------------------------------------
> +   but perhaps people would like that more?)
> +
> +With these two in place, attributes for files can be specified and
> +overridden easily.  The priority of the various files will be (from
> +lowest to highest)
> +
> + - /etc/gitconfig
> + - $HOME/.gitconfig
> + - .gitattributes
> + - $GIT_DIR/config
> +
> +The $GIT_DIR/config file is highest priority so that a user can always
> +override an attribute that is in a .gitattributes managed by someone
> +else.
> +
> +
> +== Handlers
> +
> +At this point git would be in a position to answer the question "what
> +attributes does this file have?" for any file in the tree.
> +
> +The handler system will allow us to specify operations that should be
> +performed on files with particular attributes.  For example, we might
> +want to have text files automatically filtered to match local line
> +ending conditions.
> +
> +----------------------------------------
> +[handler "fix-line-endings"]
> +  attribute = text/plain
> +  infilter = eol_lf
> +  outfilter = eol_local
> +----------------------------------------
> +
> +This definition creates a handler named "fix-line-endings" (don't suppose
> +they actually need names) that would run the "eol_lf" filter on check
> +in, and the "eol_local" filter on checkout.
> +
> +Similarly, we might want to tell git-show how to show images:
> +
> +----------------------------------------
> +[handler "show-images"]
> +  attribute = image/png
> +  attribute = image/jpeg
> +  attribute = image/gif
> +  prettyfilter = pipe display -
> +----------------------------------------
> +
> +With this filter in place,
> +----------------------------------------
> + $ git-show v1.0.0:images/picture.png
> +----------------------------------------
> +Would run "display -" and pipe the picture to it.
> +
> +Multiple filters can be supplied, and order will be important.  So, to
> +transparently decompress a file and fix the line endings for a text file
> +we might do:
> +
> +---------------------------------------
> +[handler "decompress-text"]
> +  attribute = compressed-text
> +  infilter = gunzip
> +  infilter = eol_lf
> +  outfilter = eol_local
> +  outfilter = gzip
> +---------------------------------------
> +
> +This might be useful for documentation files that come compressed (like
> +README.gz used by Debian).
> +
> +Similarly, we might want to be able to look at compressed files stored
> +in the repository:
> +
> +---------------------------------------
> +[handler "show-compressed-file"]
> +  attribute = application/gzip
> +  prettyfilter = gunzip
> +  prettyfilter = eol_auto
> +---------------------------------------
> +
> +=== Handler Types
> +
> + - infilter
> +   Run by git-add when content is added to the repository.
> + - outfilter
> +   Run by git-checkout or git-reset when copying content from the index
> +   to the working directory.  (Actually, more likely this would be done
> +   at low level in git-read-tree)
> + - prettyfilter
> +   Run by git-cat-file or git-show when content is being pretty-printed
> +   for display to the user.  If no prettyfilter is set, then it should
> +   default to outfilter.
> + - merge
> +   Git has built in, good support for merging standard text files.
> +   However, if you wanted to add an external merge tool for merging,
> +   say, XML files, you could supply it here
> + - diff
> +   If you wanted to use an external diff for certain files, you would
> +   specify it in in the diff handler
> +
> +
> +
> +=== Filter list
> +
> + - eol_lf
> +   Force all line endings to line-feed.
> +
> + - eol_cr
> +   Force all line endings to carriage-return.
> +
> + - eol_crlf
> +   Force all line endings to carriage-return, line-feed.
> +
> + - eol_local
> +   Force all line endings to local default (LF on UNIX; CRLF on Windows).
> +
> + - eol_auto
> +   Do the "right thing" depending on the file content and local default.
> +
> + - pipe <cmd>
> +   Open a bidirectional pipe and run <cmd>; which will transform the
> +   file in whatever way one wishes.
> +
> + - keywords
> +   The traditional CVS keywords are substituted in the file.  Maybe:
> +     $Id$, $Author$, $URL$, $Date$, $Rev$
> +   This filter would behave differently depending on whether it was an
> +   infilter or an outfilter.  On infilter it would remove the extra
> +   information.  On outfilter it would add the extra information.
> +
> + - gunzip, gzip / zip, unzip
> +   Some files are stored compressed in their natural state (like ODF for
> +   example).  This isn't optimum for getting the best out of git.  Git
> +   (when making pack files) can store the delta between two files, and
> +   compresses all objects in its store anyway.  So, a compressed text
> +   file that changes over time will actually take up more disk space
> +   than the uncompressed file would take up.  These compression filters
> +   would fix that problem by transparently decompressing/compressing the
> +   files on their way in and out.
A good option to have, but not a default. A problem with zip is ofcourse that it represent many files. 
Presumably 
> +
> +
> +== Problems
> +
> + - Initial Checkout
> +   The .gitattributes file is not in the working directory, so how git
> +   find out what to apply to a file?  The potential solution might be to
> +   use the in-repo .gitattributes for checking out; and the working-tree
> +   .gitattributes when checking in.
Git can read the file from the tree it is about to check out. On check in it should read
the version in the index for consitency. That still leaves the problem of which file
to check when updating the index. Should the index be recomputed when the copy
of the file in the index gets updated? Probably. The .gitattributes must be the first file
in each tree to check in order to make all other blobs consistent with it.

> + - Merge of .gitattributes
Since they are text files it shold be possible to merge, but is the format "merge friendly", i.e.
unlikely to cause conflicts and  most importantly unlikely to cause incorrect merges. The
file is not likely to be updated very often. Maybe the first case for special merge programs?

> +   What happens when .gitattributes conflicts during a merge?
Ouch, ... need more thinking 

> + - Change of .gitattributes without change of files.
Recompute index, and possible change of files in git (.e.g files with CRLF get normalized
to LF is that is the change indicated by .gitattributes).

> +   What happens when .gitattributes changes, but the files that would be
> +   affected by that change do not?  Those files really should be checked
> +   out again, to apply any new outfilter settings.

Only on git-reset/checkout.

-- robin

Re: [PATCH] Quick description of possible gitattributes system

[Junio C Hamano]

Andy Parkins <andyparkins@gmail.com> writes:

> +Attributes are attached to paths in one of two ways:
> +
> + - Via the .gitattributes file.  This will work in the same way as the
> +   .gitignore file.  That is, there can be multiple .gitattributes files
> +   throughout the working directory.
> +
> +   Typically the .gitattributes file will be a version controlled file,
> +   just like .gitignore.

Another point to stress here is that these are property of the
paths of the project and independent of the platform each user
happens to have a repository of the project on.  If something is
text, then it is text whether the platform is on UNIX or on DOS.

> +   It will obviously be of different format from the .gitattributes
> +   file, but the attribute names and path specification will be
> +   identical (and specified as if from the root of the repository tree).

I am in favor of also using $GIT_DIR/config format even for the
in-tree .gitattributes files.  For one thing, it makes the
syntax and quoting rules saner and more consistent (blanks in
pathname patterns and such), and we already have a parser for
it.

	[attribute "image/png"]
        	path = "*.png"

	[attribute "image/jpg"]
        	path = "*.jpg"
        	path = "*.jpeg"

	[attribute "text"]
		; anything that is image/* is non-text -- I am
                ; not sure if we want to allow this which means
                ; we need to resolve dependencies, though.
                attr = "!image/*"
		; inside list of path-patterns, the first match wins.
		path = "!this-is-not-text.txt"
        	path = "*"

	; .gitignore is a mere special case of "ignored" attribute.
	[attribute "ignored"]
        	path = "*~"
                path = "!hand-maintained.o"
                path = "*.[aso]"

Earlier, I said attributes are property of project, but that
does not mean we should forbid users to define more paths in
config.  On DOS, you might want to have:

	[attribute "ignored"]
        	path = "*.obj"

to suppliment the last entry of the above "ignored" that is
meant to catch object files, until the upstream project adds it
to the .gitattribute files.

In contrast to that attributes define property of paths of the
project, handlers define how we should behave on paths with
certain attributes in a particular repository.  So it is
important to stress that that handler sections should never
appear in in-tree .gitattributes files.  Because they are
per-repository, the same project checked out on DOS will be used
with this handler definition (which would be the built-in
default not to require the users to have one in ~/.gitconfig):

	[handler "text"]
        	input = crlf
                output = crlf

while on UNIX the input/output would be no-op.

> +----------------------------------------
> +[handler "fix-line-endings"]
> +  attribute = text/plain
> +  infilter = eol_lf
> +  outfilter = eol_local
> +----------------------------------------
> +
> +This definition creates a handler named "fix-line-endings" (don't suppose
> +they actually need names) that would run the "eol_lf" filter on check
> +in, and the "eol_local" filter on checkout.

I would suggest making handler section _match_ the attribute
names.  By "match", what I mean is to allow something like this:

	[handler "image/*"]
        	pretty = "cmd display -"

even though the attributes are "image/png", "image/jpg", etc.

> +[handler "show-images"]
> +  attribute = image/gif
> +  prettyfilter = pipe display -

I would recommend against calling them *filter.  After all, the
semantics of each handler action (such as input, output and
pretty) already determine if it needs to act as a filter or data
sink.  We do not need to say prettyfilter either; it is not even
a filter to begin with -- it is a data sink.

Re: [PATCH] Quick description of possible gitattributes system

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> I am in favor of also using $GIT_DIR/config format even for the
> in-tree .gitattributes files.

Thinking about this a bit more, I take this "syntax" part back.
One-line-per-record format is much nicer if we want to keep this
information in .gitattributes files in-tree, especially because
the changes to them _could_ conflict, and the pathattr mechanism
needs to work with such a file left in the working tree as its
input.  I suspect the config parser would simply say "oops" and
die given a malformed input resulting from conflict markers.  It
is easy to ignore <<< === >>> lines and we would get a sensible
result.

Which automatically means that there will be no "continuation
lines".

Re: [PATCH] Quick description of possible gitattributes system

[Andy Parkins]

On Friday 2007 March 02 00:09, Junio C Hamano wrote:

[deleted lots of easily-agreed-with stuff]

> I would suggest making handler section _match_ the attribute
> names.  By "match", what I mean is to allow something like this:

I don't see that working.  If it were possible to do that why would we need 
separate handlers and attributes?  You're example is fine

> 	[handler "image/*"]
>         	pretty = "cmd display -"

But what about when I want to do this:

[handler "lossless-images-only"]
   attribute = "image/png"
   attribute = "image/gif"
   pre-commit = /bin/false

To prevent the adding of all lossless image formats?  I actually can't see a 
time when the handler name needs referencing, so it can be anything (as long 
as it's unique - perhaps).  If you wanted to use/encourage the attribute 
name - fine, but it shouldn't be required.  However I think it's more 
sensible to name the handler after what it does than what it applies to - the 
applies to can change, what it does never.

The handlers are like function calls.  You wouldn't want to name functions 
after where they were called.

> > +[handler "show-images"]
> > +  attribute = image/gif
> > +  prettyfilter = pipe display -
>
> I would recommend against calling them *filter.  After all, the
> semantics of each handler action (such as input, output and
> pretty) already determine if it needs to act as a filter or data
> sink.  We do not need to say prettyfilter either; it is not even
> a filter to begin with -- it is a data sink.

That's true; I don't have any strong feelings.  I just didn't like "convert" 
being used.  Convert always sounds like something that takes one thing and 
makes another out of it - at the end you have two things.  Filters transform 
one thing into another - at the end you have one thing.

Perhaps better names would be simply to name them after the operation the runs 
them.

 infilter = checkin
 outfilter = checkout
 prettyfilter = show

?

Is the document useful; would you like me to maintain it?  I don't want to 
waste your time with it.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: [PATCH] Quick description of possible gitattributes system

[Andy Parkins]

On Friday 2007 March 02 04:46, Junio C Hamano wrote:

> Thinking about this a bit more, I take this "syntax" part back.
> One-line-per-record format is much nicer if we want to keep this
> information in .gitattributes files in-tree, especially because

That was my original thought.  However, it occurred to me that we might have 
an implicit handler for .gitattributes that said

[handler "attributesfile"]
  merge = builtin-attributes-merge

So you don't necessarily have to keep it line based (although I suppose that 
is easier than writing a special merger).


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: [PATCH] Quick description of possible gitattributes system

[Andy Parkins]

On Thursday 2007, March 01, Brian Gernhardt wrote:

> [attribute "image/png"]
>     path = *.png
>     show = "open %path%"
> [attribute "text/plain"]
>     path = *.c
>     checkout = eol_to_local

The problem is that this flips the relationship.  What you want to do is 
assign attributes to paths.  This is assigning paths to attributes.  
What about this:

  doc/*.txt = text/plain AND documentation

In your system that would make

[attribute text/plain]
  path = doc/*.txt
  show = "open %path%"
[attribute documentation]
  path = doc/*.txt
  show = "open %path"

i.e. you've got two sections to maintain instead of one.  That's why I 
suggested:

[attributes "doc/*.txt"]
  attributes = text/plain
  attributes = documentation

I'm not really advocating that exact syntax, but I do think it's got to 
be like having a database like this:

  paths -> path-attribute-join <- attributes
  attributes -> handlers

Your method does
  path -> attribute
  attribute -> handlers

There would therefore be excessive repetition or impossible-to-specify 
connections.


> Both of these lists look right at first glance, although I think I
> prefer something simple like "show" to "prettyfilter".  And we might
> want a "reject" or "none" for merges...  Attempting to merge a bitmap
> is madness, so the system should just output a <file>-
> <head>.<extension> or similar.

I have no major concerns about any of the syntax really.  The important thing 
is that we are clear and consistent (as always).


> > + - Change of .gitattributes without change of files.
> > +   What happens when .gitattributes changes, but the files that
> > would be
> > +   affected by that change do not?  Those files really should be
> > checked
> > +   out again, to apply any new outfilter settings.
>
> Actually, shouldn't the files also be run through the infilter to
> check for changes caused by that, too?

I don't think so.  The effect of the infilter will never be seen in the 
working tree, because it's applied on git-add.  The previous content with the 
old attributes are already in the repository.  However, it could be that we 
would have to force those files to be marked dirty in the index (this is 
already sounding bad), to force the application of the infilter on next 
checkin.  Perhaps that's what you meant, and I'm being slow.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: [PATCH] Quick description of possible gitattributes system

[Jakub Narebski]

Andy Parkins wrote:

> + - prettyfilter
> +   Run by git-cat-file or git-show when content is being pretty-printed
> +   for display to the user.  If no prettyfilter is set, then it should
> +   default to outfilter.

I'd rather have plumbing operate without filters (if it is possible),
so git-cat-file would not run prettyfilter, and git-show would run it.

Or at least detect if output is tty, so one can do something like
"git cat-file -p v1.0.0:gitweb/git-logo.png > git-logo.png" and get
expected result, while "git show v1.0.0:gitweb/git-logo.png" would
show the image.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [PATCH] Quick description of possible gitattributes system

[Linus Torvalds]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1576 bytes --]



On Fri, 2 Mar 2007, Jakub Narebski wrote:

> Andy Parkins wrote:
> 
> > + - prettyfilter
> > +   Run by git-cat-file or git-show when content is being pretty-printed
> > +   for display to the user.  If no prettyfilter is set, then it should
> > +   default to outfilter.
> 
> I'd rather have plumbing operate without filters (if it is possible),
> so git-cat-file would not run prettyfilter, and git-show would run it.

That really sucks. I do "git show xyz > filname" all the time, and while 
it's been about diffs and commits, I could imagine doing the same thing 
for things like "git show v2.6.17:Documentation/logo.gif > some.gif" too.

Yes, I know could do "git cat-file -p .." instead, but since we added "git 
show", and made it do the Right Thing (tm) for blobs, I've come to use it 
more (which is as it should be: "cat-file" is low-level plumbing, while 
"git show" is what you'd expect users to use.

So I really think that the issue here is that the *pager* should do the 
right thing. Which gets us to:

> Or at least detect if output is tty, so one can do something like
> "git cat-file -p v1.0.0:gitweb/git-logo.png > git-logo.png" and get
> expected result, while "git show v1.0.0:gitweb/git-logo.png" would
> show the image.

Yes. How about just having the built-in git pager do the right thing?

Right now we actually wait for data to start arriving (commit 35ce862) due 
to working around a bug in less. But what if we could make the pager 
actually do the right thing here automatically? With "less" just being the 
default action for *text*?

			Linus

Re: [PATCH] Quick description of possible gitattributes system

[Brian Gernhardt]

On Mar 2, 2007, at 7:00 AM, Andy Parkins wrote:

> On Thursday 2007, March 01, Brian Gernhardt wrote:
>
>> [attribute "image/png"]
>>     path = *.png
>>     show = "open %path%"
>> [attribute "text/plain"]
>>     path = *.c
>>     checkout = eol_to_local
>
> The problem is that this flips the relationship.  What you want to  
> do is
> assign attributes to paths.  This is assigning paths to attributes.
> What about this:
>
>   doc/*.txt = text/plain AND documentation
>
> In your system that would make

I'm sorry, I was assuming that information on what to do with each  
attribute would be in the config file while a majority of the  
attribute information was in an in-tree file.  I was actually assuming:

.gitattributes:
*.png: image/png
logo.png: logo
*.c: text/plain source

.git/config:
[attribute "image"]
    show = ...
    merge = ...

With the ability to have additional "path =" entries for *local*  
overrides/additions.  Storing the handler information in  
the .gitattributes is one of the worst things you could do, IMHO.  It  
assumes that people will have a homogenous environment to develop in,  
and that every developer want to use the same tools.  In some cases,  
this may be true (eol_lf on checking for everybody), but not in  
others.  On my Mac, I'd like to use "open" for random file types, and  
might want to use VERY different graphical merge than Linux users.   
And OS X is a fairly Unix-like environment.  Let's not even ponder  
those poor fools using MinGW.  ;-)

>> Actually, shouldn't the files also be run through the infilter to
>> check for changes caused by that, too?
>
> I don't think so.  The effect of the infilter will never be seen in  
> the
> working tree, because it's applied on git-add.  The previous  
> content with the
> old attributes are already in the repository.  However, it could be  
> that we
> would have to force those files to be marked dirty in the index  
> (this is
> already sounding bad), to force the application of the infilter on  
> next
> checkin.  Perhaps that's what you meant, and I'm being slow.

That's something like what I meant.  And it sounds bad.  Content  
changes caused by attribute changes should likely be handled by the  
user and made easy to detect instead of trying to handle it  
automatically.  So ignore my original statement on the matter.

~~ Brian

Re: [PATCH] Quick description of possible gitattributes system

[Andy Parkins]

On Friday 2007, March 02, Brian Gernhardt wrote:

> I'm sorry, I was assuming that information on what to do with each
> attribute would be in the config file while a majority of the
> attribute information was in an in-tree file.  I was actually
> assuming:

It's definitely in the .gitattributes file, but is also optionally in 
the config as well.  This means that a user can always override 
something locally - i.e. they're always in control of their own 
repository, even when they disagree with upstream (perhaps).

> .git/config:
> [attribute "image"]
>     show = ...
>     merge = ...
>
> With the ability to have additional "path =" entries for *local*
> overrides/additions.  Storing the handler information in

That's almost exactly it; but it makes the assumption that each 
attribute will have one unique handler.  Separating them means that 
multiple attributes can use one handler (or set of handlers).

> the .gitattributes is one of the worst things you could do, IMHO.  It
> assumes that people will have a homogenous environment to develop in,

Oh definitely.  This is one thing that everybody absolutely agrees on.  
The in-tree file /only/ adds attributes and never says what effect 
those attributes have.



Andy

-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: [PATCH] Quick description of possible gitattributes system

[Andy Parkins]

On Friday 2007, March 02, Linus Torvalds wrote:

> Yes. How about just having the built-in git pager do the right thing?

Perfect.  This is absolutely the right thing to do I think.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: [PATCH] Quick description of possible gitattributes system

[Brian Gernhardt]

On Mar 2, 2007, at 2:35 PM, Andy Parkins wrote:

>> .git/config:
>> [attribute "image"]
>>     show = ...
>>     merge = ...
>>
>> With the ability to have additional "path =" entries for *local*
>> overrides/additions.  Storing the handler information in
>
> That's almost exactly it; but it makes the assumption that each
> attribute will have one unique handler.  Separating them means that
> multiple attributes can use one handler (or set of handlers).

That's why I suggested keeping the handler section, but not tying  
handlers to attributes in the attribute section.  I think it follows  
the principle of least surprise to be able to do as I demonstrated  
above, but allow reuse as I described originally (handler option in  
the attribute section).  My major concern is how do you tell what  
order handlers are run in for a given file if the handlers specify  
the attributes rather than the other way around?  And why make the  
user do things like:

[handler "text"]
    attribute=text
    ...

Instead of:

[attribute "text"]
    ...

(The handler section could be called "mangle", "blah", or "why-do-I- 
have-to-name-this", and the point would be the same.)  If you want to  
run the same handlers/filters/whatever for multiple types, create a  
handler section and reference it from all of those types or give all  
the files a new attribute that explains why you'd want to handle them  
identically and put the options on that attribute instead.

The attribute is the important thing here, not the handlers, so they  
should be primary, not secondary.  And that's my major point that I'm  
trying to get around to I guess.  I don't care about naming handler  
sections, searching through the file to find their order, or anything  
other than *what git is doing to my files*.  The files (content,  
actually) are king.

~~ Brian

Re: [PATCH] Quick description of possible gitattributes system

[Linus Torvalds]

On Fri, 2 Mar 2007, Andy Parkins wrote:

> On Friday 2007, March 02, Linus Torvalds wrote:
> 
> > Yes. How about just having the built-in git pager do the right thing?
> 
> Perfect.  This is absolutely the right thing to do I think.

Well, it would be perfect, except it's rather hard to do. Right now we 
simply don't have any way to tell the pager what to do with the data, and 
we'd need to do some communications passing thing to let it know.

It *could* just look at the data directly, but that's actually hard: if we 
start looking at the data, there's no way to push the data back onto the 
head of a pipe, and there's no really good way to tell an external pager 
to "start off with this data that I already read earlier to figure out the 
type, and then continue with that other file descriptor that I'm passing 
in".

So then we'd have to have a totally useless process in between the git 
process and the external pager to just feed the data from one pipe to the 
other..

Doing the pager internally would obviously solve that issue, but I really 
don't think we want to do that, especially since it's very 
system-dependent. And temporary files suck for all the other reasons 
(incrementally generated data).

So we're in the situation where:

 - the pager process *will* wait until actual data is starting to appear,
   so we *can* have some side-band channel to tell it "oh, btw, if there 
   is a pager, this is going to be image data, so start up an external 
   image viewer instead". 

 - but I don't have a good clue what side-band to use. We could use 
   a special "FILE *pagerdata", of course (which would just be fd#3 in 
   the pager). Then, "git show" could just do something like

	if (pager_in_use)
		fprintf(pagerdata, "'%s'\n", type);

   and we could change pager.c to do something like the appended patch.

but I have to say, it looks a bit strange.

		Linus

---
diff --git a/pager.c b/pager.c
index 5f280ab..b71dd44 100644
--- a/pager.c
+++ b/pager.c
@@ -9,6 +9,8 @@
 
 static void run_pager(const char *pager)
 {
+	static char input_type[100] = "text";
+
 	/*
 	 * Work around bug in "less" by not starting it until we
 	 * have real input
@@ -17,7 +19,16 @@ static void run_pager(const char *pager)
 
 	FD_ZERO(&in);
 	FD_SET(0, &in);
-	select(1, &in, NULL, &in, NULL);
+	FD_SET(3, &in);
+	select(4, &in, NULL, &in, NULL);
+	if (FS_ISSET(3, &in)) {
+		int n = read(3, input_type, sizeof(input_type)-1);
+		if (n > 0)
+			input_type[n] = 0;
+	}
+	close(3);
+
+	pager = select_pager(input_type, pager);
 
 	execlp(pager, pager, NULL);
 	execl("/bin/sh", "sh", "-c", pager, NULL);

Re: [PATCH] Quick description of possible gitattributes system

[Johannes Schindelin]

Hi,

On Fri, 2 Mar 2007, Linus Torvalds wrote:

> Doing the pager internally would obviously solve that issue,

Ooooooh! A builtin! (Seriously, I thought about that already, but 
dismissed it for the reasons you said, and more...)

> So we're in the situation where:
> 
>  - the pager process *will* wait until actual data is starting to appear,
>    so we *can* have some side-band channel to tell it "oh, btw, if there 
>    is a pager, this is going to be image data, so start up an external 
>    image viewer instead". 
> 
>  - but I don't have a good clue what side-band to use. We could use 
>    a special "FILE *pagerdata", of course (which would just be fd#3 in 
>    the pager). Then, "git show" could just do something like
> 
> 	if (pager_in_use)
> 		fprintf(pagerdata, "'%s'\n", type);
> 
>    and we could change pager.c to do something like the appended patch.
> 
> but I have to say, it looks a bit strange.

The obvious thing, of course, is a git-show specific solution (this is no 
patch, it only illustrates my idea):

 builtin-log.c |   11 +++++++++++
 git.c         |    2 +-
 2 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/builtin-log.c b/builtin-log.c
index 1c9f7d0..83e6cea 100644
--- a/builtin-log.c
+++ b/builtin-log.c
@@ -123,6 +123,7 @@ int cmd_show(int argc, const char **argv, const char *prefix)
 	struct rev_info rev;
 	struct object_array_entry *objects;
 	int i, count, ret = 0;
+	enum { NO_PAGER, TEXT_PAGER, OTHER_PAGER } pager_mode = NO_PAGER;
 
 	git_config(git_log_config);
 	init_revisions(&rev, prefix);
@@ -140,6 +141,16 @@ int cmd_show(int argc, const char **argv, const char *prefix)
 	for (i = 0; i < count && !ret; i++) {
 		struct object *o = objects[i].item;
 		const char *name = objects[i].name;
+		if (o->type == OBJ_BLOB) {
+			/* determine if name falls into some special class */
+			if (pager_mode == NO_PAGER &&
+					setup_special_pager(name))
+				pager_mode = OTHER_PAGER;
+		}
+		if (pager_mode == NO_PAGER) {
+			setup_pager();
+			pager_mode = TEXT_PAGER;
+		}
 		switch (o->type) {
 		case OBJ_BLOB:
 			ret = show_object(o->sha1, 0);
diff --git a/git.c b/git.c
index 909adf2..a3811e6 100644
--- a/git.c
+++ b/git.c
@@ -291,7 +291,7 @@ static void handle_internal_command(int argc, const char **argv, char **envp)
 		{ "runstatus", cmd_runstatus, RUN_SETUP | DISALLOW_IN_GIT_DIR },
 		{ "shortlog", cmd_shortlog, RUN_SETUP | USE_PAGER },
 		{ "show-branch", cmd_show_branch, RUN_SETUP },
-		{ "show", cmd_show, RUN_SETUP | USE_PAGER },
+		{ "show", cmd_show, RUN_SETUP },
 		{ "stripspace", cmd_stripspace },
 		{ "symbolic-ref", cmd_symbolic_ref, RUN_SETUP },
 		{ "tar-tree", cmd_tar_tree },

Re: [PATCH] Quick description of possible gitattributes system

[Andy Parkins]

On Friday 2007, March 02, Linus Torvalds wrote:
> On Fri, 2 Mar 2007, Andy Parkins wrote:
> > On Friday 2007, March 02, Linus Torvalds wrote:
> > > Yes. How about just having the built-in git pager do the right
> > > thing?
> >
> > Perfect.  This is absolutely the right thing to do I think.
>
> Well, it would be perfect, except it's rather hard to do. Right now
> we simply don't have any way to tell the pager what to do with the
> data, and we'd need to do some communications passing thing to let it
> know.

There is an alternative.  pager.c:run_pager() runs a select() to wait 
for data before it actually exec()s the pager.  What if after the 
select() the process didn't exec(), but read the first line to decide 
what pager to exec?

Then, when pager_in_use is true, the sending process writes, say, a 
handler identifier which tells the pager process what to exec().



Andy

-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: [PATCH] Quick description of possible gitattributes system

[Andy Parkins]

On Friday 2007, March 02, Andy Parkins wrote:

> There is an alternative.  pager.c:run_pager() runs a select() to wait
> for data before it actually exec()s the pager.  What if after the
> select() the process didn't exec(), but read the first line to decide
> what pager to exec?

Duh!  I see now that that is almost exactly what you did in your patch.  
Ignore me.


Andy

-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: [PATCH] Quick description of possible gitattributes system

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Fri, 2 Mar 2007, Jakub Narebski wrote:
>
>> Andy Parkins wrote:
>> 
>> > + - prettyfilter
>> > +   Run by git-cat-file or git-show when content is being pretty-printed
>> > +   for display to the user.  If no prettyfilter is set, then it should
>> > +   default to outfilter.
>> 
>> I'd rather have plumbing operate without filters (if it is possible),
>> so git-cat-file would not run prettyfilter, and git-show would run it.
>
> That really sucks. I do "git show xyz > filname" all the time, and while 
> it's been about diffs and commits, I could imagine doing the same thing 
> for things like "git show v2.6.17:Documentation/logo.gif > some.gif" too.
>
> Yes, I know could do "git cat-file -p .." instead, but since we added "git 
> show", and made it do the Right Thing (tm) for blobs, I've come to use it 
> more (which is as it should be: "cat-file" is low-level plumbing, while 
> "git show" is what you'd expect users to use.
>
> So I really think that the issue here is that the *pager* should do the 
> right thing...

Sorry, but I do not think that would work because of the way we
setup our pager.  The original process turns into the PAGER and
child ships the data to it.

	$ git show -s master maint :t/test4012.png

I do not particularly think the prettyfilter is useful in
practice (it was just a fun toy I did to convince me that I can
use it for things other than input/output munging), but at least
we could make it a bit more usable by this, on top of 'pu':

diff --git a/builtin-log.c b/builtin-log.c
index 86062d3..bc26358 100644
--- a/builtin-log.c
+++ b/builtin-log.c
@@ -116,7 +116,7 @@ static int show_object(const unsigned char *sha1, const char *path, int nohead)
 	if (size <= offset)
 		goto finish;
 
-	if (path) {
+	if (pager_in_use && path) {
 		const struct pathattr *a = pathattr_lookup(path);
 		if (a && a->pretty) {
 			pretty = a->pretty;

Re: [PATCH] Quick description of possible gitattributes system

[Jakub Narebski]

On Fri, 2 Mar 2007, Linus Torvalds wrote:
> On Fri, 2 Mar 2007, Jakub Narebski wrote:
>> Andy Parkins wrote:
>> 
>>> + - prettyfilter
>>> +   Run by git-cat-file or git-show when content is being pretty-printed
>>> +   for display to the user.  If no prettyfilter is set, then it should
>>> +   default to outfilter.
>> 
>> I'd rather have plumbing operate without filters (if it is possible),
>> so git-cat-file would not run prettyfilter, and git-show would run it.
[...]
> So I really think that the issue here is that the *pager* should do the 
> right thing. Which gets us to:
> 
>> Or at least detect if output is tty, so one can do something like
>> "git cat-file -p v1.0.0:gitweb/git-logo.png > git-logo.png" and get
>> expected result, while "git show v1.0.0:gitweb/git-logo.png" would
>> show the image.
> 
> Yes. How about just having the built-in git pager do the right thing?
> 
> Right now we actually wait for data to start arriving (commit 35ce862) due 
> to working around a bug in less. But what if we could make the pager 
> actually do the right thing here automatically? With "less" just being the 
> default action for *text*?

Wouldn't it be better and simplier instead of creating built-in magic
(as in DWIM) pager, to have in place of "prettyfilter", "pretty",
"show" handler "pager" handler, which would define _default_ pager for
given attributes.

It would mean that the filter/sink would be run only when pager is run,
i.e. for "git show" and "git -p cat-file"/"git --paginate cat-file".
I guess that would mean that handler action would be run only for tty.
And you could always override it by setting PAGER environmental variable.

So it would be:

+ - pager
+   Change default pager for this content. If it is not set, it defaults
+   to `less'. You can always override default pager by setting PAGER
+   environmental variable.

-- 
Jakub Narebski
ShadeHawk on #git
Poland