GIT PULL] ext4 updates for 3.5

GIT PULL] ext4 updates for 3.5

[Theodore Ts'o]

Sorry this is so late... it's been a crazily busy May...

	      	 	      	   - Ted


The following changes since commit 69964ea4c7b68c9399f7977aa5b9aa6539a6a98a:

  Linux 3.4-rc5 (2012-04-29 15:19:10 -0700)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git dev

for you to fetch changes up to 5e44f8c374dc4f8eadf61cd18b2c0d46bc87c1b7:

  ext4: hole-punch use truncate_pagecache_range (2012-06-01 00:15:28 -0400)

----------------------------------------------------------------
Ext4 updates for 3.5

The major new feature added in this update is Darrick J. Wong's
metadata checksum feature, which adds crc32 checksums to ext4's
metadata fields.  There is also the usual set of cleanups and bug
fixes.

----------------------------------------------------------------
Akira Fujita (1):
      ext4: remove needs_recovery in ext4_mb_init()

Andreas Dilger (1):
      ext4: disallow hard-linked directory in ext4_lookup

Dan Carpenter (1):
      ext4: fix potential NULL dereference in ext4_free_inodes_counts()

Darrick J. Wong (23):
      ext4: create a new BH_Verified flag to avoid unnecessary metadata validation
      ext4: change on-disk layout to support extended metadata checksumming
      ext4: record the checksum algorithm in use in the superblock
      ext4: load the crc32c driver if necessary
      ext4: calculate and verify superblock checksum
      ext4: calculate and verify inode checksums
      ext4: calculate and verify checksums for inode bitmaps
      ext4: calculate and verify block bitmap checksum
      ext4: verify and calculate checksums for extent tree  blocks
      ext4: Calculate and verify checksums for htree nodes
      ext4: calculate and verify checksums of directory leaf blocks
      ext4: Calculate and verify checksums of extended attribute blocks
      ext4: make block group checksums use metadata_csum algorithm
      ext4: add checksums to the MMP block
      jbd2: change disk layout for metadata checksumming
      jbd2: enable journal clients to enable v2 checksumming
      jbd2: Grab a reference to the crc32c driver if necessary
      jbd2: checksum journal superblock
      jbd2: checksum revocation blocks
      jbd2: checksum descriptor blocks
      jbd2: checksum commit blocks
      jbd2: checksum data blocks that are stored in the journal
      ext4/jbd2: add metadata checksumming to the list of supported features

Eric Sandeen (1):
      ext4: force ro mount if ext4_setup_super() fails

Haogang Chen (1):
      ext4: fix potential integer overflow in alloc_flex_gd()

Hugh Dickins (1):
      ext4: hole-punch use truncate_pagecache_range

Salman Qazi (2):
      ext4: add ext4_mb_unload_buddy in the error path
      ext4: remove mb_groups before tearing down the buddy_cache

Tao Ma (2):
      ext4: protect group inode free counting with group lock
      ext4: let getattr report the right blocks in delalloc+bigalloc

Theodore Ts'o (7):
      ext4: remove unnecessary check in add_dirent_to_buf()
      ext4: enable the 64-bit jbd2 feature based on the 64-bit ext4 feature
      ext4: remove redundundant "(char *) bh->b_data" casts
      ext4: return ENOMEM when mounts fail due to lack of memory
      ext4: add debugging trigger for ext4_error()
      ext4: add missing save_error_info() to ext4_error()
      ext4: don't trash state flags in EXT4_IOC_SETFLAGS

Wanlong Gao (1):
      jbd2: use kmem_cache_zalloc wrapper instead of flag

Zheng Liu (3):
      ext4: cleanup in ext4_discard_allocated_blocks()
      ext4: fix format flag in ext4_ext_binsearch_idx()
      ext4: use consistent ssize_t type in ext4_file_write()

 fs/ext4/Kconfig            |   2 +
 fs/ext4/balloc.c           |  41 ++++-
 fs/ext4/bitmap.c           |  83 +++++++++
 fs/ext4/dir.c              |  12 ++
 fs/ext4/ext4.h             | 130 +++++++++++--
 fs/ext4/ext4_extents.h     |  24 +++
 fs/ext4/ext4_jbd2.c        |   9 +-
 fs/ext4/ext4_jbd2.h        |   7 +-
 fs/ext4/extents.c          |  91 +++++++--
 fs/ext4/file.c             |   2 +-
 fs/ext4/ialloc.c           |  81 +++++++--
 fs/ext4/inode.c            | 119 ++++++++++--
 fs/ext4/ioctl.c            |  19 +-
 fs/ext4/mballoc.c          |  30 +--
 fs/ext4/mmp.c              |  44 ++++-
 fs/ext4/namei.c            | 445 ++++++++++++++++++++++++++++++++++++++++++---
 fs/ext4/resize.c           |  71 +++++++-
 fs/ext4/super.c            | 253 +++++++++++++++++++++-----
 fs/ext4/xattr.c            |  92 ++++++++--
 fs/ext4/xattr.h            |   4 +-
 fs/jbd2/Kconfig            |   2 +
 fs/jbd2/commit.c           |  70 ++++++-
 fs/jbd2/journal.c          | 132 +++++++++++++-
 fs/jbd2/recovery.c         | 126 ++++++++++++-
 fs/jbd2/revoke.c           |  27 ++-
 fs/jbd2/transaction.c      |   4 +-
 include/linux/jbd2.h       |  59 +++++-
 include/linux/jbd_common.h |   2 +
 28 files changed, 1784 insertions(+), 197 deletions(-)

Re: GIT PULL] ext4 updates for 3.5

[Linus Torvalds]

On Fri, Jun 1, 2012 at 8:07 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>
>  git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git dev

Hmm, this is a regular branch, but you *usually* use a tag, and the
format of the pull-request imples that too:

> ----------------------------------------------------------------
> Ext4 updates for 3.5
>
> The major new feature added in this update is Darrick J. Wong's
> metadata checksum feature, which adds crc32 checksums to ext4's
> metadata fields.  There is also the usual set of cleanups and bug
> fixes.
>
> ----------------------------------------------------------------

Is there perhaps a tag I should pull instead?

               Linus

Re: GIT PULL] ext4 updates for 3.5

[Linus Torvalds]

On Fri, Jun 1, 2012 at 8:07 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>
>  git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git dev

Hmm, this is a regular branch, but you *usually* use a tag, and the
format of the pull-request imples that too:

> ----------------------------------------------------------------
> Ext4 updates for 3.5
>
> The major new feature added in this update is Darrick J. Wong's
> metadata checksum feature, which adds crc32 checksums to ext4's
> metadata fields.  There is also the usual set of cleanups and bug
> fixes.
>
> ----------------------------------------------------------------

Is there perhaps a tag I should pull instead?

               Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: GIT PULL] ext4 updates for 3.5

[Mark Lord]

On 12-06-01 11:07 AM, Theodore Ts'o wrote:
> Ext4 updates for 3.5
> 
> The major new feature added in this update is Darrick J. Wong's
> metadata checksum feature, which adds crc32 checksums to ext4's
> metadata fields.  There is also the usual set of cleanups and bug
> fixes.

That sounds vaguely like an on-disk change that may have issues
with systems that regularly switch between older and newer kernel versions.

Could you elaborate on it a bit more, thanks!

Re: GIT PULL] ext4 updates for 3.5

[Linus Torvalds]

On Fri, Jun 1, 2012 at 8:44 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> Is there perhaps a tag I should pull instead?

Never mind, I see it. It's 'ext4_for_linus'.

So here's the odd 'git' trick of the day.  Do

   git ls-remote
git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4 | less -S

to see all the branches and tags. In there, we see the branch you
asked me to pull:

  5e44f8c374.. refs/heads/dev

and then we can see if perhaps any of the tags have that same commit
when peeled. So search for 5e44f8c374, and sure enough:

  ..
  4e4d525149..  refs/tags/ext4_for_linus
  5e44f8c374..  refs/tags/ext4_for_linus^{}
  ..

so that 'ext4_for_linus' tag (which obviously has a different SHA1
than the commit it points to) points to the same commit (the "^{}" is
just the git SHA1 expression for "commit pointed to by object").

So pulling 'ext4_for_linus' instead of 'dev' will get me the same
commits you asked me to pull, with the added tag information that your
pull request containes.

Now I just wonder why your pull request said 'dev' at all.

            Linus

Re: GIT PULL] ext4 updates for 3.5

[Ted Ts'o]

On Fri, Jun 01, 2012 at 10:19:33AM -0700, Linus Torvalds wrote:
> So pulling 'ext4_for_linus' instead of 'dev' will get me the same
> commits you asked me to pull, with the added tag information that your
> pull request containes.

So I should do "git request-pull origin git://... ext4_for_linus" next
time, right?

Sorry about that.  I thought it had included the reference to tag
automatically since the text of the tag showed in the request-pull
text.

> Now I just wonder why your pull request said 'dev' at all.

Because that was the name of the branch head that I used; sorry about
that.

							- Ted

Re: GIT PULL] ext4 updates for 3.5

[Ted Ts'o]

On Fri, Jun 01, 2012 at 12:45:27PM -0400, Mark Lord wrote:
> On 12-06-01 11:07 AM, Theodore Ts'o wrote:
> > Ext4 updates for 3.5
> > 
> > The major new feature added in this update is Darrick J. Wong's
> > metadata checksum feature, which adds crc32 checksums to ext4's
> > metadata fields.  There is also the usual set of cleanups and bug
> > fixes.
> 
> That sounds vaguely like an on-disk change that may have issues
> with systems that regularly switch between older and newer kernel versions.
> 
> Could you elaborate on it a bit more, thanks!
> 

It's an on-disk change, but it's gated by a superblock "feature flag".
So unless you actually activate the feature, you won't get it.  If you
do activate the feature, then you won't be able to switch between
older and newer kernel versions --- at least not and be able to mount
the file system read/write.  (We have different feature flags that
indicate whether or not the kernel is allowed to mount the file system
read/write, read/only, or not at all, if it doesn't know about a bit
in the COMPAT, COMPAT_RO, or INCOMPAT feature bitmak.)

The e2fsprogs support for this feature is currently only in the
(rewinding) proposed update branch, so it's not something that I
recommend people use just yet.  Even though it's been pretty well
tested, there are probably still some bugs we still need to shake out.

	      	  	   	      	      	    - Ted

Re: GIT PULL] ext4 updates for 3.5

[Linus Torvalds]

On Fri, Jun 1, 2012 at 10:43 AM, Ted Ts'o <tytso@mit.edu> wrote:
>
> So I should do "git request-pull origin git://... ext4_for_linus" next
> time, right?

Right.

> Sorry about that.  I thought it had included the reference to tag
> automatically since the text of the tag showed in the request-pull
> text.

Apparently git request-pull is too damn smart for its own good, and
will find the tags even when they aren't mentioned, making you think
it made it to me. Bot it doesn't - so then I have to go search for
them by hand.

Added Junio and git to the cc: if git finds the tag and uses that for
the request-pull message text, then it should use the tagname for the
pull request line too. Or, alternatively, don't be smart at all, and
force users to name the tag explicitly. The current "both smart and
dumb" combination seems to be the worst of both world.

              Linus

Re: GIT PULL] ext4 updates for 3.5

[Linus Torvalds]

On Fri, Jun 1, 2012 at 10:43 AM, Ted Ts'o <tytso@mit.edu> wrote:
>
> So I should do "git request-pull origin git://... ext4_for_linus" next
> time, right?

Right.

> Sorry about that.  I thought it had included the reference to tag
> automatically since the text of the tag showed in the request-pull
> text.

Apparently git request-pull is too damn smart for its own good, and
will find the tags even when they aren't mentioned, making you think
it made it to me. Bot it doesn't - so then I have to go search for
them by hand.

Added Junio and git to the cc: if git finds the tag and uses that for
the request-pull message text, then it should use the tagname for the
pull request line too. Or, alternatively, don't be smart at all, and
force users to name the tag explicitly. The current "both smart and
dumb" combination seems to be the worst of both world.

              Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: GIT PULL] ext4 updates for 3.5

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> Apparently git request-pull is too damn smart for its own good, and
> will find the tags even when they aren't mentioned, making you think
> it made it to me. Bot it doesn't - so then I have to go search for
> them by hand.

Something like this, perhaps.

 git-request-pull.sh | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/git-request-pull.sh b/git-request-pull.sh
index e6438e2..5ca786e 100755
--- a/git-request-pull.sh
+++ b/git-request-pull.sh
@@ -57,9 +57,13 @@ headrev=$(git rev-parse --verify "$head"^0) || exit
 merge_base=$(git merge-base $baserev $headrev) ||
 die "fatal: No commits in common between $base and $head"
 
-# $head is the token given from the command line. If a ref with that
-# name exists at the remote and their values match, we should use it.
-# Otherwise find a ref that matches $headrev.
+# $head is the token given from the command line, and $tag_name, if
+# exists, is the tag we are going to show the commit information for.
+# If that tag exists at the remote and it points at the commit, use it.
+# Otherwise, if a branch with the same name as $head exists at the remote
+# and their values match, use that instead.
+#
+# Otherwise find a random ref that matches $headrev.
 find_matching_ref='
 	sub abbr {
 		my $ref = shift;
@@ -70,24 +74,29 @@ find_matching_ref='
 		}
 	}
 
-	my ($exact, $found);
+	my ($tagged, $branch, $found);
 	while (<STDIN>) {
 		my ($sha1, $ref, $deref) = /^(\S+)\s+(\S+?)(\^\{\})?$/;
 		next unless ($sha1 eq $ARGV[1]);
 		$found = abbr($ref);
+		if ($deref) {
+			$tagged = $found;
+			last;
+		}
 		if ($ref =~ m|/\Q$ARGV[0]\E$|) {
 			$exact = $found;
-			last;
 		}
 	}
-	if ($exact) {
+	if ($tagged) {
+		print "$tagged\n";
+	} elsif ($exact) {
 		print "$exact\n";
 	} elsif ($found) {
 		print "$found\n";
 	}
 '
 
-ref=$(git ls-remote "$url" | perl -e "$find_matching_ref" "$head" "$headrev")
+ref=$(git ls-remote "$url" | perl -e "$find_matching_ref" "$head" "$headrev" "$tag_name")
 
 url=$(git ls-remote --get-url "$url")
 

Re: GIT PULL] ext4 updates for 3.5

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> +# $head is the token given from the command line, and $tag_name, if
> +# exists, is the tag we are going to show the commit information for.
> +# If that tag exists at the remote and it points at the commit, use it.
> +# Otherwise, if a branch with the same name as $head exists at the remote
> +# and their values match, use that instead.
> +#
> +# Otherwise find a random ref that matches $headrev.

It may also be a good idea to add something like:

	if test -n "$tag_name" && test "tags/$tag_name" != "$ref"
        then
		echo "warning: have you really pushed the tag yet???"
	fi

somewhere near we show "cat-file tag $tag_name" information in the
output.

Re: GIT PULL] ext4 updates for 3.5

[Linus Torvalds]

On Fri, Jun 1, 2012 at 12:00 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
> It may also be a good idea to add something like:
>
>        if test -n "$tag_name" && test "tags/$tag_name" != "$ref"
>        then
>                echo "warning: have you really pushed the tag yet???"
>        fi
>
> somewhere near we show "cat-file tag $tag_name" information in the
> output.

Ack.

          Linus

Re: GIT PULL] ext4 updates for 3.5

[Ted Ts'o]

One other thought.  One of the reasons why I used "dev" and not
"ext4_for_linus" in my git request-pull command line was because that
several months ago I had created a branch called ext4_for_linus, and I
hadn't gotten around to deleting it, so I figured I'd just use "dev"
instead, since it was non-ambiguous.

If there is a case where there is both tag and a branch with the same
name, it might be a good idea for request-pull not just to issue a
warning, since the user might not notice the warning, but to just
abort --- and make the user explicitly type refs/tags/ext4_for_linus
or refs/heads/ext4_for_linus.

Most of the time if the user ignores the warning about the ambiguity,
it's not such a big deal, but in the case of a pull-request, he or she
is going to be possibly embarassing themself publically, so it might
be better to explicitly list the reason why a non-qualified refname is
ambiguous and force the user to pick the right one.

It's a minor point, and usually I double check the commit id in the
pull request just to be sure, so it might be considered too much of
coddling the user, but it might be a good safety check to add.

						- Ted

Re: GIT PULL] ext4 updates for 3.5

[Linus Torvalds]

On Fri, Jun 1, 2012 at 1:38 PM, Ted Ts'o <tytso@mit.edu> wrote:
> One other thought.  One of the reasons why I used "dev" and not
> "ext4_for_linus" in my git request-pull command line was because that
> several months ago I had created a branch called ext4_for_linus, and I
> hadn't gotten around to deleting it, so I figured I'd just use "dev"
> instead, since it was non-ambiguous.

You can always just use "tags/xyzzy" to disambiguate..

                  Linus

Re: GIT PULL] ext4 updates for 3.5

[Ted Ts'o]

On Fri, Jun 01, 2012 at 01:43:35PM -0700, Linus Torvalds wrote:
> On Fri, Jun 1, 2012 at 1:38 PM, Ted Ts'o <tytso@mit.edu> wrote:
> > One other thought.  One of the reasons why I used "dev" and not
> > "ext4_for_linus" in my git request-pull command line was because that
> > several months ago I had created a branch called ext4_for_linus, and I
> > hadn't gotten around to deleting it, so I figured I'd just use "dev"
> > instead, since it was non-ambiguous.
> 
> You can always just use "tags/xyzzy" to disambiguate..

Ah, I thought you always had to type the full "refs/tags/.."
specification.  I didn't realize it did substring matching.  Besides,
"dev" was much shorter and easier to type.  :-)

One of the things which is easy for a novice to miss is the "warning:
ext4_for_linus is ambiguous" message, though.  And I suspect many
novices don't know about the "git show-refs -a | grep ambiguous_tag"
trick to figure out where the ambiguity is coming from.  So if we are
going to give a warning, it might be good if we actually told the user
what were the other choices, and which choice git was going to choose.
Most of the time it seems to choose the right thing, but trusting
computer programs to choose the right answer from something ambiguous
via some hueristic always scares the bejesus out of me....

    	 	   	  	     	     - Ted

Re: GIT PULL] ext4 updates for 3.5

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Fri, Jun 1, 2012 at 1:38 PM, Ted Ts'o <tytso@mit.edu> wrote:
>> One other thought.  One of the reasons why I used "dev" and not
>> "ext4_for_linus" in my git request-pull command line was because that
>> several months ago I had created a branch called ext4_for_linus, and I
>> hadn't gotten around to deleting it, so I figured I'd just use "dev"
>> instead, since it was non-ambiguous.
>
> You can always just use "tags/xyzzy" to disambiguate..

Not only that, the "look at the ls-remote output and guess what you
are teeling the recipient to pull" code gives tags/ext4_for_linus
even if there is not ext4-for-linus branch; if I recall correctly we
did this on purpose so that older git that did not DWIM "git pull
ext4_for_linus" to look at targs/ext4_for_linus.